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SUMMARY 


1 . PROBLEM 

The  purpose  of  this  study  was  to  assess  the  effective- 
ness, utility  and  acceptability  of  a prototype  human 
resources  (HR)  data  handbook  (Reed,  Snyder,  Bar an,  Loy  and 
Curtin,  1975)  developed  by  the  Air  Force  Human  Resources 
Laboratory.  HR  data  are  data  describing  what  people  can 
do,  how  much  they  cost,  their  availability,  and  the  need 
for  them.  A handbook  of  such  data  is  needed  by  system 
development  personnel  because  these  data  significantly 
influence  the  effectiveness  and  cost  of  the  system  and 
because  available  information  is  widely  scattered  through- 
out many  sources . 

Once  such  a handbook  is  compiled,  the  question  is 
whether'  it  will  be  effective  and  useful  in  terms  of  solving 
relevant  problems,  and  just  as  important,  whether  it  will 
be  accepted  by  its  eventual  user.  The  user  is  generally 
someone  untrained  in  a behavioral  orientation.  Hence,  it 
is  necessary  to  discover  whether  the  material  presented  to 
him  has  been  translated  into  his  terms  and  whether  he  will 
make  use  of  the  handbook. 

Such  an  assessment  process  is  particularly  necessary 
in  the  case  of  the  prototype  handbook  because,  although'' 
there  have  been  other  behavioral  data  handbooks  of  a 
molecular,  human  engineering  character,  the  prototype 
handbook  is  the  first  that  deals  with  human  resources  and 
behavior  on  a molar  level. 

2 . APPROACH 

Since  the  prototype  handbook  is  compiled  to  be  used  in 
a particular  context  (in  this  case,  system  development) , 
the  most  appropriate  methodology  would  be  to  give  it  to 
potential  users  to  employ  the  handbook  in  relevant  work 
activities  and  then  ask  them  to  evaluate  the  prototype 
handbook. 

This  approach  runs  into  practical  difficulties,  e.g., 
time  required,  limited  observational  opportunities,  a -small 
user  population,  the  possibility  of  interference  with  on- 
going development  activities.  The  solution  adopted  was  to 
obtain  data  from  typical  users  during  and  after  their  use  of 
the  handbook  to  solve  hypothetical  or  simulated  system 
development  problems . 

Twelve  representative  problems  based  on  prototype 
handbook  parameters  were  included  in  a questionnaire  format. 
Six  of  the  problems  compared  a new  system  of  indexing  data 
(the  Master  Index)  with  the  more  familiar  Alphabetical 
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Index.  In  these  indexing  problems,  test  participants 
(henceforth  to  be  termed  "participants,"  "respondents," 
"users"  or  "personnel"  for  short)  were  required  to  enter 
the  prototype  handbook  and  determine,  by  means  of  the 
indexing  system  specified,  which  handbook  table  would 
provide  appropriate  data.  In  these  indexing  problems, 
participants  did  not  have  to  solve  the  problems. 

In  the  remaining  six  data  extraction  problems,  the  page 
number  of  the  correct  table  was  given  the  participant  along 
with  the  problem  and  he  was  required  to  use  the  table  to 
solve  the  problem.  In  two  of  these  problems,  personnel 
were  required  to  solve  the  problem  first  without,  and  then 
with,  the  aid  of  the  prototype  handbook,  thus  permitting  a 
comparison  of  solution  efficiency  with  and  without  the 
handbook . 

Following  each  data  extraction  problem  the  user  was 
asked  to  rate  such  factors  as  the  difficulty  level  of  tne 
problem,  his  confidence  in  the  correctness  of  his  answer, 
the  similarity  of  the  problem  presented  to  the  ones  he 
customarily  encountered,  and  the  usefulness  of  the  prototype 
handbook  data  in  solving  the  problem. 

Following  all  12  problems,  participants  were  asked  to 
rate  the  utility  of  the  prototype  handbook  as  a whole. 

Rating  instruments  were  5-point  Likert-type  scales. 

Three  criteria  were  applied  to  the  assessment: 
handbook  effectiveness  (the  user's  ability  to  solve  problems 
correctly  and  in  reasonable  time) ; handbook  utility 
(whether  handbook  data  were  useful  in  solving  problems); 
and  handbook  acceptability  (ratings  of  handbook  usefulness 
and  influence  in  system  design) . 

Participants  came  from  two  populations:  10  from  System 

Program  Offices  (SPO)  at  Wright-Patterson  Air  Force  Base 
(WPAFB) , Ohio;  26  from  the  F-15  design  organization  at  the 
McDonnell-Douglas  Astronautics  Company  in  St.  Louis.  This 
was  to  determine  whether  the  prototype  handbook  was  equally 
useful  to  SPO  and  contractor  personnel.  In  both  groups, 
participants  varied  widely  in  terms  of  specialty  areas  and 
years  of  experience. 

The  questionnaire  was  administered  first  at  Wright- 
Patterson  Air  Force  Base  and  subsequently  at  McDonnell- 
Douglas.  No  changes  in  the  questionnaire  between  the  first 
and  second  administration  were  necessary.  At  WPAFB, 
respondents  were  given  the  handbook  and  the  questionnaire 
and  allowed  to  complete  the  form  ad  libitum.  At  McDonnell- 
Douglas,  they  were  issued  the  handbook  and  then  assigned  a 
scheduled  meeting  time  during  which  to  complete  the 
questionnai re . 


Data  were  analyzed  by  both  parametric  and  non- 
parametric  methods  (the  latter  to  check  on  the  possibility 
that  the  non-normal  distribution  of  responses  might  distort 
the  analysis ) . 

3.  RESULTS 

Although  participants  had  little  or  no  opportunity  to 
study  the  prototype  handbook  before  using  it  to  solve 
questionnaire  problems,  they  solved  68%  correctly.  Solution 
efficiency  varied  greatly  among  personnel:  25-92%  correct 

(mean  64%)  in  the  WPAFB  sample;  46-92%  correct  (mean  70%) 
in  the  McDonnell-Douglas  sample.  Differences  between  the 
WPAFB  and  McDonnell-Douglas  samples  were  not  significant 
and  consequently  the  two  were  combined. 

Average  task  completion  or  solution  time  was  around 
fiVe  minutes  per  problem,  although  the  range  was  much 
broader  than  that  for  individual  problems. 

Personnel  were  significantly  more  correct  in  solving 
problems  with  than  without  the  aid  of  the  prototype  hand- 
book. On  one  of  the  two  comparison  problems,  respondents 
felt  that  the  problem  solved  without  the  prototype  handbook 
was  significantly  more  difficult  than  the  same  problem 
solved  with  the  handbook.  They  also  had  significantly  more 
confidence  on  problems  solved  with  the  aid  of  the  prototype 
handbook  than  without  that  aid. 

The  perceived  difficulty  of  the  problems  presented  was 
rated  at  3.6,  very  close  to  the  indifference  point  (4.0). 
Indexing  problems  appeared  *~o  be  significantly  more 
difficult  than  data  extract  jn  problems.  Participants 
found  the  task  of  discovering  the  correct  table  number  to 
be  greater  than  that  of  using  the  material  in  the  prototype 
handbook.  On  the  other  hand,  they  were  more  efficient  with 
indexing  problems  than  they  were  with  data  extraction 
problems . 

Forty-four  percent  of  the  participants  considered  the 
prototype  handbook  as  having  moderate  or  greater  utility  and 
62%  as  having  potentially  moderate  or  greater  influence  c?n 
their  design  work.  They  felt  that  their  own  data>  sources 
were  just  as  useful  or  only  slightly  less  useful  than  the 
prototype  handbook.  On  the  other  hand,  participants  felt 
that  their  own  data  sources  were  less  accessible  than  the 
prototype  handbook. 

Those  who  felt  the  prototype  handbook  had  utility  and 
influence  on  design  represented  a broad  range  of  engineering 
backgrounds,  although  concentrated  in  specialties  that 
customarily  require  more  analysis  of  personnel  interfaces 
(e.g.,  Human  Factors,  Maintainability,  Reliability, 
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Operations  Analysis)  than  does  general  design  engineering. 
Those  who  had  worxed  on  problems  similar  to  those  for  which 
the  prototype  handbook  was  developed  were  more  positive 
about  the  handbook. 

Participants  preferred  to  use  the  Master  Index  system 
of  indexing  in  preference  to  the  Alphabetical  Index,  but 
performed  more  effectively  with  the  latter,  presumably 
because  they  were  more  familiar  with  it.  They  also 
considered  that  the  Master  Index  system  was  more  difficult 
to  use  than  the  Alphabetical  Index,  in  part  because  the 
problems  that  had  to  be  solved  with  the  Master  Index  were 
more  difficult  than  those  with  the  Alphabetical  Index. 

Participants  recommended  a number  of  improvements  to 
the  prototype  handbook,  including  updating  of  data  and 
making  the  indexing  systems  easier  to  use. 

4.  CONCLUSIONS 

* 

The  results  of  this  assessment  study  suggest  the 
following  conclusions: 

a.  System  development  personnel  can  use  the  prototype 
handbook  to  make  significantly  more  correct  decisions  than 
without  the  handbook.  Engineers  have  greater  confidence  in 
decisions  made  with  the  prototype  handbook  than  without  it. 

b.  If  one  considers  that  a substantial  percentage  of 
participants  viewed  the  prototype  handbook  as  having 
utility  and  potential  influence  on  design,  the  audience  for 
this  handbook  is  potentially  large.  Engineers  had  some 
difficulty  recognizing  the  kinds  of  problems  for  which  the 
prototype  handbook  was  designed  as  ones  they  ordinarily 
encountered,  but  considered  these  problems  to  be  realistic. 
Engineers  consider  their  own  data  sources  almost  as  good  as 
the  prototype  handbook,  but  much  less  accessible. 

c.  Those  who  saw  utility  in  the  prototype  handbook  and 
are  hence  more  likely  to  use  it  are  more  likely  to  have 
specialized  jobs  (e.g..  Human  Factors,  Maintainability,  Crew 
Station  Design)  than  general  design  functions.  Those  who 
have  worked  on  problems  of  the  type  dealt  with  by  the 
prototype  handbook  tended  to  be  more  positive  to  that 
handbook  and  are  therefore  more  likely  to  use  it. 

d.  Engineers  preferred  to  use  the  Master  Index  system 
in  preference  to  the  Alphabetical  Index,  but  experienced 
greater  difficulty  and  performed  less  effectively  with  the 
former.  The  Master  Index  system  can  be  used  to  facilitate 
prototype  handbook  use,  but  should  be  refined. 
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e.  A number  of  improvements  were  recommended  by 
assessment  participants,  including  updating  the  data, 
simplifying  the  Master  Index  system,  reducing  verbiage  in 
the  tables  and  clarifying  the  implications  of  the  prototype 
handbook  data.  These  improvements  are  required  to  make  the 
prototype  handbook  maximally  useful. 

The  assessment  was  sufficiently  positive  to  warrant 
continuing  efforts  to  develop  handbooks  and/or  data  banks 
of  HR  data  for  use  in  the  design  of  new  systems  and 
equipments . 
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ASSESSMENT  OF  A PROTOTYPE  HUMAN  RESOURCES 
DATA  HANDBOOK  FOR  SYSTEMS  ENGINEERING 

I.  BACKGROUND  AND  PURPOSE 


BACKGROUND 

This  study  describes  the  assessment  of  a prototype 
handbook  of  human  resources  (HR)  data  (Reed,  Snyder,  Baran, 
Loy,  & Curtin,  1975)  developed  by  the  Air  Force  Human 
Resources  Laboratory. 

Before  describing  this  handbook,  the  purpose  of  its 
evaluation  and  the  methodology  employed,  it  will  be  helpful 
to  provide  the  reader  with  the  background  of  this  study. 

Repeated  attempts  have  been  made  to  provide  system 
development  personnel  (primarily  design  engineers  but  also 
including  human  factors  personnel)  with  compilations  of 
behavioral  data.  One  of  the  earliest  was  the  justly  famous 
"Human  Engineering  Guide  for  Equipment  Designers"  (Woodson, 
1954,  Woodson  & Conover,  1964) , which  was  followed  by  a 
spate  of  handbooks  for  various  purposes  (e.g.,  Parker  & 

West,  1973)  . These,  however,  dealt  with  the  relatively 
molecular  aspects  of  man-machine  performance,  those 
subsumed  under  the  rubric  of  human  factors  engineering. 

The  prototype  handbook  evaluated  in  this  study  is  the  first, 
however,  that  deals  with  the  more  molar  considerations  of 
HR,  which  has  been  defined  Askren  (1973)  in  the  following 
manner:  "Human  resources  data  are  those  data  which  describe 

the  people  of  an  organization  in  terms  of  what  they  can 
contribute,  how  much  they  cost,  how  available  they  are,  how 
perishable  they  are,  and  how  many  of  them  are  needed." 

Why  is  it  necessary  to  assess  a handbook  of  HR  data 
and  how  is  that  assessment  accomplished? 

Rarely  is  the  desirability  of  developing  handbooks 
questioned.  HR  research  which  is  directed  at  the  solution 
of  urgent  system  development,  training  and  operational 
problems  can  have  little  value  unless  it  is  communicated  to 
those  who  need  the  information.  Since  HR  data  are  scattered 
among  a great  variety  of  sources,  they  are  not  readily 
available  to  the  engineer  or  human  factors  specialist  when 
needed.  Without  a single  source  the  HR  specialist  has 
difficulty  in  finding  and  consolidating  information  into 
usable  form. 

The  questions  that  do  arise  are  very  pragmatic  ones: 
What  information  should  be  provided  in  a handbook;  how 
should  that  information  be  presented  for  optimal  use?  If 
doubts  are  raised  about  the  utility  of  a particular 
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handbook,  these  are  not  about  the  desirability  of  handbooks 
per  se  but  reflect  the  possibility  that  the  handbook  in 
question  may  not  provide  the  proper  information,  that  the 
information  may  not  be  directed  to  the  appropriate  "target" 
audience  or  that  the  method  of  presentation  may  not  be 
optimal.  Because  of  these  possibilities,  it  becomes 
necessary  to  assess  the  effectiveness  with  which  the 
handbook  accomplishes  its  purpose. 

The  assessment  of  HR  handbooks  centers  around  three 
questions : 

(1)  Can  the  intended  audience  for  such  a handbook 
(e.g.,  engineers,  human  factors  specialists)  use  the 
handbook?  That  is,  are  the  data  phrased  and  presented  in 
terms  such  that  the  intended  audience  can  utilize  the  data 
to  solve  system  development  problems?  Are  the  data  relevant 
to  the  kinds  of  system  development  problems  the  intended 
audience  usually  encounters? 

(2)  Will  the  intended  audience  use  the  handbook? 
Engineers  are  not  notorious  for  willingly  applying 
behavioral  data  and  incorporating  behavioral  inputs  into 
development.  Even  the  most  popular  handbook  (Woodson  & 
Conover,  1964)  has  only  a comparatively  small  audience 
among  engineers.  One  may  question  whether  readership  in 
the  conventional  sense  of  newspaper,  magazine  or  novel 
circulation  should  be  one  of  the  criteria  of  behavioral 
handbook  utility,  since  the  solution  of  system  development 
problems  is  not  ordinarily  based  on  popularity.  Use  of  a 
handbook  in  even  a few  critical  system  development  instances 
would  presumably  more  than  compensate  for  the  labor,  time 
and  money  required  to  produce  that  handbook.  Nevertheless, 
one  does  wish  to  know  what  the  likelihood  is  that  certain 
data  compilations  will  be  used.  No  definitive  answer  to  the 
question  can  be  given,  and  any  answer  is  at  best  a highly 
subjective  estimate. 

(3)  What  handbook  improvements  are  necessary  and 
desirable?  Rarely  does  the  first  version  of  a handbook  not 
need  revision. 

The  first  two  questions  have  significant  philosophical 
and  methodological  implications,  as  we  shall  see  later.  For 
one  thing,  they  establish  the  criteria  to  be  employed  in 
performing  the  handbook  assessment. 

Our  concern  for  determining  whether  its  intended 
audience  can  use  the  prototype  handbook  to  solve  system 
development  problems  stems  from  the  fact  that  the  audience 
for  HR  data  handbooks  is  composed,  in  addition  to  behavioral 
specialists  themselves,  of  system  planners,  design  and 
system  engineers,  operations  analysts,  reliability/ 
maintainability  engineers,  etc.  It  is  necessary,  therefore, 
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that  special  efforts  be  made  to  ensure  that  these  users 
understand  the  data  provided  them. 

Obviously,  the  information  transmitted  by  the  HR 
researcher  must  be  understood  by  the  latter.  Here  we 
encounter  a formidable  difficulty  because  the  engineer  has 
been  trained  in  ,a  physical,  not  a behavioral  framework. 

From  this  arises  the  problem  of  translating  behavioral 
information  into  engineering  language. 

Attempts  have  been  made  to  determine  those  characteris- 
tics which  behavioral  data  should  have  to  appeal  to 
engineers  (Devoe,  1963,  Sinaiko,  1963,  Meister  & Farr,  1967). 
Among  these  are  heavy  emphasis  on  quantitative  data, 
graphics  and  tables.  Alternative  information  formats  have’ 
been  exposed  to  engineers  to  determine  their  preferences 
(see  Meister,  1971,  for  a review  of  such  studies) . 

The  necessity  for  translating  inputs  into  engineering 
terms  (whatever  these  are,  and  the  concept  is  very  vague) 
assumes  that  the  engineer  will  be  the  prime  manipulator  of 
these  inputs.  The  initiator  of  the  HR  input-  the  human 
factors  specialist-  is  not  ordinarily  in  a position  to 
create  his  own  design;  he  must  work  through  the  engineer. 
One  of  the  concepts  behind  the  development  of  behavioral 
handbooks  for  engineers  is  the  idea  that  engineers  should 
ultimately  assume  as  many  personnel  design  functions  as 
possible . 

Efficient  integration  of  HR  inputs  into  system 
development  assumes  that  (1)  the  system  development 
personnel  are  provided  with  required  information;  (2)  they 
are  willing  to  utilize  that  information. 

The  first  assumption  presupposes  that  (a)  the  informa- 
tion is  provided  to  the  proper  personnel,  and  (b)  the 
information  is  appropriate  to  the  system  development 
problem  it  addresses.  Since  system  development  is  very 
complex,  it  has  become  highly  specialized  (e.g., 
reliability /maintainability , system,  electronics,  avionics, 
training,  etc.,  engineers).  It  is  possible  that  information 
addressing  particular  system  development  problems  may  be 
given  to  personnel  who  face  entirely  different  problems. 
Handbook  developers  and  assessors  must  therefore  ask 
themselves:  Just  who  will  read  and  apply  this  handbook; 

what  problems  should  the  handbook  address;  is  the 
information  appropriate  to  these  problems? 

The  second  assumption — the  willingness  of  system 
development  personnel  to  use  HR  information — poses  more 
serious  problems  because  it  is  influenced  by  many  factors 
not  under  the  handbook  developer's  control.  These  factors 
include  the  engineer's  personality,  training,  experiential 
background  and  attitudes  toward  behavior,  behavioral 
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science  and  HR  specialists.  Obviously,  if  a handbook's 
intended  audience  refuses  to  accept  the  material  presented, 
handbook  utility  is  nsopardized. 

It  is  often  assumed  that  if  the  translation  of  HR 
inputs  into  engineering  terms  is  accomplished,  system 
developers  will  automatically  endeavor  to  use  these  inputs. 
This  is  a logical  assumption  which  may  be  negated  by  the 
idiosyncratic  factors  previously  referred  to. 

Because  the  engineer  is  mainly  responsible  for 
incorporating  HR  inputs  into  design,  he  becomes  the  ultimate 
standard  of  the  utility  of  HR  data.  It  therefore  becomes 
necessary  to  expose  the  prototype  handbook  to  the  engineer's 
own  assessment. 

Underlying  efforts  to  persuade  engineers  to  utilize  HR 
data  are  a number  of  assumptions  that  should  be  examined 
critically  by  handbook  developers  (although  this  report  does 
not  pretend  to  do  so  in  any  depth) . 

1.  System  development  personnel  need  HR  data  in  order 
to  perform  tradeoff  analyses  that  involve  personnel.  No 
one  can  logically  quarrel  with  this  concept. 

2.  The  kind  of  HR  data  needed  by  system  development 
personnel  and  the  manner  in  which  these  data  should  be 
presented  has  been  ascertained.  To  a certain  extent  data 
communication  requirements  for  engineers  have  been  fairly 
well  established  (graphics,  numerics,  simplicity,  precision) 
and  there  have  been  attempts  to  determine  the  precise 
behavioral  information  the  design  engineer  needs  (Hannah  & 
Reed,  1965;  Meister,  Sullivan,  Finley  & Askren,  1969a, 
1969b) . However,  given  the  different  backgrounds  system 
development  personnel  have  and  the  variety  of  development 
jobs  they  perform,  it  is  difficult  to  match  up  the  types  of 
personnel  with  their  specific  needs  for  HR  data. 

3.  Since  the  system/design  engineer  is  the  one  who 
applies  HR  data  to  developmental  decisions,  his  ability  and 
willingness  to  use  these  data  become  the  standard  against 
which  one  assesses  the  adequacy  of  these  data.  The  present 
study  was  performed  on  the  basis  of  that  assumption. 

This  is  not  an  unreasonable  standard.  It  is,  however, 
affected  by  the  engineer’s  training  and  idiosyncracies . 

Use  of  the  standard  requires  one  also  to  consider  the 
implications  of  the  engineer's  failure  to  accept  HR  data. 

If  these  data  are  not  accepted  by  their  intended  users,  are 
they  invalid  or  irrelevant? 
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HR  data  collected  under  the  proper  conditions  can  never 
be  invalid  in  the  sense  of  being  unrepresentative  of  the 
conditions  under  which  they  were  gathered,  but  they  can  be 
irrelevant  in  terms  of  the  uses  to  which  one  wishes  to  put 
them.  Different  types  of  behavioral  data  are  required  for 
different  purposes..  For  example,  human  engineering  data 
are  one  distinct  variety  used  for  making  bench-level  design 
decisions,  and  there  are  some  excellent  human  engineering 
handbooks  (e.g. , Woodson  & Conover,  1964;  Van  Cott  & 
Kinkade,  1975) . It  is  possible  that  user  reaction  to  these 
data  determines  whether  the  data  are  directed  to  the 
correct  audience  and  deal  with  the  appropriate  questions. 

On  the  other  hand,  reliance  on  positive  user  reaction 
to  behavioral  data  as  the  criterion  of  data  relevance  means 
that  the  behavioral  specialist's  responsibility  for 
determining  the  meaningfulness  of  his  inputs  to  development 
is  substantially  transferred  to  the  engineer.  With  the 
user's  generally  low  level  of  application  of  behavioral 
inputs,  is  it  realistic  to  allow  his  reaction  to  determine 
what  inputs  are  given  him?  Ultimately  one  must  ask  whether 
non-behavioral  personnel  are  qualified  to  sit  in  judgment 
on  thgse  inputs. 

The  following  questions  must  therefore  be  addressed  by 
handbook  developers; 

1.  Who  should  be  the  primary  audience  for  behavioral 
data  handbooks?  The  behavioral  specialist  who  then 
translates  these  data  to  other  system  development  personnel? 
If  so,  the  audience  for  the  handbook  is  small  and  highly 
specialized.  Should  it  be  other  system  development 
personnel  for  whom  the  behavioral  handbook  seeks  to  perform 
the  translation  of  behavioral  inputs  into  engineering  terms 
directly?  In  that  event,  what  then  becomes  the  role  of  tne 
human  factors  specialist  in  system  development?  Should  the 
handbook  address  both  types  of  audience?  Can  one  format 
data  for  two  such  different  types  of  personnel  in  the  same 
handbook? 

2.  Of  system  development  personnel  other  than  human 
factors  specialists,  is  there  a subset  - those  with 
distinctive  backgrounds  (engineering  specialties  such  as 
maintainability,  crew  system  design,  etc.)  - to  whom  the 
behavioral  handbook  should  be  directed?  In  other  words, 
how  broad  should  the  targeted  audience  for  the  handbook  be? 

3.  What  kind  of  reaction  signifies  that  the  intended 
audience  will  in  fact  make  use  of  the  behavioral  handbook? 
What  does  the  developer  expect  the  user  to  do  with  the 
handbook? 

With  these  last  questions  we  approach  the  problems 
involved  in  assessing  the  prototype  handbook. 
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DESCRIPTION  OF  THE  HR  HANDBOOK 


Although  it  is  impossible  to  present  complete  details 
of  the  prototype  handbook,  it  is  necessary  for  the  reader's 
understanding  of  this  report  to  describe  the  outline  of  that 
handbook.  The  following  material  is  taken  (with  some  slight 
modifications)  directly  from  the  handbook  (Reed  et  al,  1975). 

The  prototype  handbook  was  prepared:  (1)  to  assess  the 

need  for  a technical  reference  containing  HR  data,  and 
(2)  to  determine  whether  it  was  feasible  to  combine  HR  data 
obtained  from  many  different  sources  into  a format  that 
conveyed  meaningful  information  to  potential  users.  The 
intention  was  to  develop  a limited  handbook  containing 
samples  of  data  that  must  be  given  consideration  in  a full- 
scale  handbook  development  program.  Since  this  prototype 
handbook  contains  only  a limited  amount  of  data,  the  data 
do  not  present  a complete  picture  of  Air  Force  human 
resources  utilization. 

The  prototype  handbook  contains  data  on  manpower, 
personnel  skills,  training,  maintenance  performance, 
logistics  and  costs  as  they  relate  to,  and  interact  with, 
operational  systems  and  subsystems.  Potential  users  of  the 
prototype  handbook  include  human  factors  specialists, 
training  planners,  design  engineers,  configuration  managers, 
system  planners,  etc.  The  intent  was  to: 

1.  Consolidate  HR  data  applicable  to  system  design  and 
development.  These  data  are  usually  scattered  in  many 
government  and  contractor  data  banks,  technical  reports, 
operational  commands,  and  in  the  form  of  expert  opinion. 

2.  Assist  the  user  to  determine  how  human  resources 
are  influenced  by  system  design  and  vice  versa. 

3.  Provide  a means  by  which  specialists  involved  in 
system  design  and  development  can  make  optimal  use  of  HR 
data . 

4.  Present  data  from  different  disciplines. 

5.  Provide  a means  by  which  design  problems  can  be 
identified  and  resolved.  The  prototype  handbook  should 
allow  design  tradeoff  decisions  to  take  into  account  the 
constraints  of  the  Air  Force’s  human  resources. 

The  prototype  handbook  is  limited  to  data  on  the 
functions  performed  by  the  avionics  career  field  on  the 
fire  control  system  of  nine  fighter  systems.  These  systems 
include  the  F-106A/B,  F-105D,  F-4C,  F-4D,  F-4E,  F-lllA, 
FB-111A,  A-7D,  and  the  F-15.  Naturally  the  prototype 
handbook  does  not  contain  all  of  the  information  that  would 
be  included  in  a full-scale  handbook. 
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The  prototype  handbook  is  organized  into  three  data 
sections  and  two  indexing  schemes.  The  three  data  sections 
are : 

Section  I - Empirical  data  on  fire  control  systems.  In 
general,  this  section  contains  comparisons  between  system 
designs,  training,  support  manpower,  occupational  jobs/tasks, 
maintenance  procedures,  logistic  support  and  various  costs. 

Section  II  - Human  resources  posture.  This  section 
contains  information  pertinent  to  past,  current  and 
projected  numbers  of  personnel  with  various  skill  and 
experience  levels. 

Section  III  - General  references.  Included  in  this  section 
are  data  on  the  effects  of  task  complexity,  the  time 
required  for  Air  Force  maintenance  personnel  to  acquire 
certain  skills,  error  rates  in  performing  maintenance 
activities,  performance  time,  etc.  With  few  exceptions, 
each  data  page  in  the  prototype  handbook  contains  a set  of 
relationships  that  stands  alone  and  is  relatively  self- 
explanatory.  The  upper  third  of  the  data  page  presents  a 
set  of  functions,  a table,  function  flow,  bar  chart  or  other 
form  showing  the  pertinent  data  relationships.  The  contents 
of  each  table,  shown  in  Figure  1,  are: 

1.  Title. 

2.  Comments:  A short  description  of  the  data,  the 

methods  used  to  collect  the  data,  population  sampled, 
definitions  or  other  pertinent  information. 

3.  Implications:  A short  summary  of  the  possible 

implications  of  the  data  to  Air  Force  system  design, 
development,  operations,  etc. 

4.  Data  Sources:  A list  of  references  from  which  the 

data  were  obtained. 

5.  Models  for  Data  Application:  An  index  number (s)  to 

a mathematical  model (s)  contained  in  Section  III. 

6.  Subject:  A short  title  of  data  contents  with  an 

emphasis  on  the  key  words  appearing  in  the  Master  Index 
Tables  of  Content. 

7.  Index:  The  two  major  index  numbers  keyed  to  the 

Master  Index. 

8.  Cross-index:  A reference  to  a data  page  containing 

related  information. 

9.  Page  Numbers:  The  page  numbers  are  keyed  to  the 

indexing  scheme  of  this  handbook. 
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Nil.!.:  lie  Id  L ion  of  3AHR32231  Training  Time  to  Fire  Control  Radar  Subsystem 

Acquisition  Cost 

COMMENTS:  Course  3A3R32231  provides  formal  school  training^  for  Semi-Skilled 

Level  3 Mechanics  cn  Fire  Control  racar  subsystems,  using  equipment  accuisition 
cost?  as  a single  logical  descriptor  of  tecnnological  sopnlsticaticn  (see  Chart 
1. 3-2.1), its  relaticns.nip  to  training  time  was  examinee. 

IMPLICATIONS:  Data  failed  to  yield  significant  findings.  However,  there  was  a 

moderate  negative  relationship,  i.e.,  a tendency  for  training  tire  to  decrease  as 
subsystem  cost  increased. 

DATA  SOURCES:  1.  ATC/ACMF,  Randolph  A^3,  Texas,  October  1972. 

2.  USAF  Logistics  Command,  Wrignt-Fatterson  AF3. 


MODELS  FOR 
DATA  APPLICATION: 

III. 7-42. 1(1) 


SUBJECT: 

Training  Time  vs.  Fire  Control 
Radar  Subsystem  Cost 


CROSS  INDEX:  I.  3-2.1 


Figure  1.  Representative  handbook  data 


There  are  two  procedures  which  can  be  followed  to  gain 
access  to  needed  data.  The  first  procedure  (Master  Index, 
or  MI  as  it  is  abbreviated  in  this  report)  allows  the  reader 
to  search  for  specific  data  relationships.  If  he  is 
interested  in  determining  the  relationship  which  may  exist 
between  maintenance  manhours  and  certain  subsystems,  he  must 
turn  to  the  MI  and  the  MI  Tables  of  Content.  In  its  present 
form  the  MI  was  developed  specifically  for  the  prototype 
handbook  because  it  presumably  allows  more  efficient  entry, 
permitting  the  user  to  go  directly  to  the  relevant  data. 

The  second  (more  conventional)  procedure  (Alphabetic  Index 
or  AI)  provides  the  user  with  an  alphabetical  listing  of 
major  topic  areas  covered  in  the  handbook.  This  listing  is 
located  in  the  back  portion  of  the  handbook. 

The  index  number  which  identifies  a table  is  composed 
of  four  numbered  elements,  each  separated  by  a dash  or 
period  (e.g.,  1.11-9.5).  The  first  element  is  a Roman 
numeral  that  identifies  the  section  of  the  handbook.  The 
next  two  elements  in  Arabic  numerals  are  keyed  to  the  Master 
Index.  The  first  of  these  two  elements  is  drawn  from  the 
numbers  in  the  left-hand  vertical  margin  of  the  Master  Index 
and  the  second  from  the  diagonal  margin.  (See  Figures  2 and 
3.)  The  last  element,  also  in  Arabic  numerals,  is  keyed  to 
the  Master  Index  Tables  of  Content. 

The  procedures  for  finding  needed  data  in  the  prototype 
handbook  require  three  steps  illustrated  in  Figure  2.  Say 
that  the  user  would  like  to  know  if  there  is  a difference 
in  maintenance  manhours  (MMH ) for  removal  actions 
of  fire  control  radar  subsystems  on  the  various  fighter 
systems.  First,  the  user  must  turn  to  the  MI  to  determine 
whether  this  type  of  information  exists  in  the  prototype 
handbook.  By  scanning  the  left-hand  margin  he  finds  that 
maintenance  manhour  information  is  in  Section  I under  the 
general  subtitle,  Operations , and  has  the  index  number  11. 
The  user  then  matches  this  index  number  with  one  provided  in 
the  diagonal  margin.  For  maintenance  removal  actions,  the 
appropriate  index  number  is  9.  Thus,  the  appropriate  cell 
in  the  Master  Index  is  11-9.  The  next  step  is  to  proceed  to 
the  MI  Tables  of  Content  located  immediately  following  the 
MI  page.  The  index  numbers  at  the  top  of  each  table  are  in 
sequential  order  and  match  those  in  the  cells  of  the  MI. 

In  the  example  provided  here,  the  user  seeks  Table  11-9  for 
Section  I data  and  finds  that  the  needed  information  is  on 
Page  1.11-9.5.  The  last  numeral  of  the  index  is  found  under 
the  appropriate  column  of  the  table. 

The  procedures  above  can  be  summarized  as  follows: 

1.  Determine,  the  section  number  in  which  the  needed 
data  may  be  found. 
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2.  Find  the  appropriate  index  number  from  the  Master 
Index. 

3.  Deterito  ae  the  page  number  from  the  Master  Index  , ] 

Tables  of  Content. 

PURPOSE  OF  THE  STUDY 

This  study,  designed  to  assess  the  prototype  handbook, 
had  several  purposes:  To  determine 

1.  The  acceptability  and  usability  of  this  type  of 
handbook  among  system  development  specialists; 

2.  The  acceptability  of  the  data  presentation  and 
indexing; 

3.  The  possible  impact  of  the  handbook  concept  on 
system  design  decisions. 

4.  A last,  but  by  no  means  less  important,  purpose 
was  to  suggest  ways  of  improving  the  prototype  handbook. 
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II.  METHODOLOGY 


INTRODUCTION 

The  general  methodology  for  assessing  the  prototype 
handbook  had  several  introductory  steps  which  involved 
determination  of: 

1.  The  general  assessment  methodology; 

2.  The  assessment  criteria  to  be  employed; 

3.  The  format  of  the  assessment  instrument; 

4.  The  type  of  user/participant  to  whom  the  assessment 
instrument  was  to  be  administered. 

Before  proceeding  further,  it  is  important  to  consider 
these  factors. 

GENERAL  ASSESSMENT  METHODOLOGY 

A prototype  handbook  can  be  assessed  in  two  ways. 

One  can  submit  the  handbook  to  a reviewer  who  need  do  no 
more  than  read  the  material  and  present  his  opinion  about 
its  adequacy.  Essentially  this  is  the  way  most  textbooks 
and  novels  are  assessed. 

Alternatively,  since  the  prototype  handbook,  unlike  a 
novel,  was  designed  to  be  used  in  a particular  context 
(system  development) , one  could  give  the  handbook  to 
potential  users  and  ask  them  to  employ  it  in  their  customary 
work  activities. 

The  latter  would  be  ideal,  but  unfortunately  it  runs 
into  practical  difficulties.  Opportunities  for  using  the 
prototype  handbook  in  actual  system  development  might  occur 
only  erratically;  therefore,  the  time  involved  in  gathering 
the  desired  information  would  probably  be  prolonged.  The 
difficulties  of  extracting  the  necessary  assessment 
l information  from  on-going  activities  might  make  heavy 

demands  on  users  as  well  as  possibly  hindering  actual 
development.  Only  a relatively  few  users  could  be  tapped 
for  their  responses. 

An  alternative  solution  would  be  to  simulate  in  paper- 
and-pencil  mode  the  kinds  of  system  development  activities 
in  which  the  prototype  handbook  would  be  used;  and  to 
require  personnel  to  use  the  handbook  in  pei forming  these 
simulated  activities. 
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This  procedure,  which  was  followed  by  Meister  & Farr 
(1965)  in  a previous  study  of  designer  reactions  to  human 
factors  inputs,  was  employed  also  for  this  study.  Since 
the  prototype  handbook  was  designed  to  be  used  in  solving 
system  development  tradeoff  problems  involving  personnel 
(as  well  as  other  parameters) , a series  of  such  problems 
was  devised.  To  make  use  of  the  prototype  handbook,  these 
problems  had  to  reflect  the  parameters  and  data  available 
in  the  handbook.  The  system  design/tradeoff  situation  was 
simulated  by  a series  of  miniature  design  "cases"  which 
presumably  called  upon  the  behaviors  the  participant  would 
employ  in  the  actual  situation.  Such  problems  might  take 
the  following  general  form: 

Assume  you  are  a system  designer  who 
is  faced  with  a number  of  alternative 
means  of  implementing  the  system 
(which  are  then  described) . The 
system  has  such  and  such  characteris- 
tics (e.g.,  varying  levels  of 
automation) . What  degree  of 
automation  will  enable  you  to  reduce 
the  required  personnel  skill  level 
and  at  what  level  will  savings 
based  on  reduced  skill  equal  the 
increased  cost  of  automation? 

One  may  ask  whether  the  developmental  situations 
simulated  are  indeed  representative  of  those  found  in 
actual  development.  If  they  are  not,  then  even  if  the 
prototype  handbook  can  be  used  effectively  in  the 
assessment  setting,  it  will  not  necessarily  be  effective 
in  actual  development. 

The  question  of  problem  "fidelity"  creates  method- 
ological difficulties  for  the  assessment.  Obviously,  the 
assessment  problems  must  reflect  handbook  parameters.  One 
would  not,  for  example,  provide  an  encyclopedia  of  church 
history  as  an  aid  in  solving  system  development  problems. 

On  the  other  hand,  the  prototype  handbook  does  not  deal 
with  all  types  of  system  development  problems;  in 
particular  it  does  not  provide  data  related  to  molecular, 
human  engineering  (e.g.,  "knobs  and  dials")  problems. 

One  might  of  course  ask  participants  whether  the 
problems  presented  to  them  are  those  they  ordinarily 
encounter  during  system  development.  If  they  answer  yes, 
then  the  question  of  problem  fidelity  is  answered 
satisfactorily.  Suppose,  however,  they  answer  no,  or 
rarely.  Are  the  prototype  handbook  parameters  as 
reflected  by  these  problems  then  invalid?  With  a broad 
spectrum  of  users  and  the  fact  that  system  development 
presents  them  with  problems  at  varying  levels  of  complexity, 
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one  would  expect  that  some  users  would  never  or  only  rarely 
have  encountered  these  problems. 

The  parameters  in  the  prototype  handbook  describe 
empirical  data  relationships  derived  (largely)  from  Air 
Force  records.  Consequently,  they  are  inherent  in  system 
development,  although  any  particular  group  of  system 
development  personnel  might  not  be  required  to  deal  with 
these  parameters.  Indeed,  without  a source  such  as  the 
prototype  handbook  the  opportunity  to  deal  with  such 
parameters  probably  would  not  be  available.  Engineers 
might  be  aware,  for  example,  of  the  question  of  whether 
enough  personnel  of  the  specified  type  would  be  available 
for  their  new  system;  but  this  question  ordinarily  would 
not  be  presented  to  them  as  a problem  to  be  solved  because 
means  (i.e.,  the  prototype  handbook)  of  answering  the 
question  or  of  including  the  factor  of  personnel  avail- 
ability in  design  considerations  would  not  be  available  to 
them.  One  must  therefore  rely  ultimately  for  the  answers 
to  the  question  of  handbook  validity  on  what  has  been 
termed  in  other  contexts  "construct  validity." 

ASSESSMENT  CRITERIA 

The  questions  posed  by  the  assessment  require  various 
criteria.  The  question,"  can  system  development  personnel 
use  the  prototype  handbook?'  requires  effectiveness  and 
utility  criteria;  the  question,  "will  they  use  the  handbook?" 
requires  criteria  of  acceptability  to  users.  The  three 
criteria  can  be  measured  both  objectively  and  subjectively. 

1.  Prototype  handbook  effectiveness  is  defined  by  the 
participant's  employment  of  the  data  presented  to  derive 
meaningful  decisions.  He  might  of  course  arrive  at  a 
decision  with  no  aid  but  his  own  experience;  that,  after 
all,  is  the  method  many  system  development  personnel  employ. 
If  the  prototype  handbook  is  efficient,  however,  the 
decision  the  user  arrives  at  with  its  aid  should  be 
substantially  more  effective-  more  precise.-  more  accurate- 
than  the  one  he  achieves  without  the  handbook.  He  should 
be  able  to  arrive  at  a decision  in  a reasonable  period  of 
time.  He  should  have  more  confidence  in  a decision  based 
on  the  prototype  handbook;  a problem  solved  with  the  aid 
of  the  prototype  handbook  should  seem  less  difficult  than 
the  same  problem  solved  without  the  handbook.  In  order  to 
demonstrate  these  points  two  things  are  necessary:  (1)  any 

problem  requiring  a design  decision  must  have  an  objectively 
correct  response  (as  defined  by  the  data  in  the  prototype 
handbook) , with  other  responses  being  either  incorrect  or 
less  correct  than  the  one  specified;  (2)  users  should  have 
the  opportunity  to  solve  design  trade-off  problems  with  and 
without  the  HR  handbook. 
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In  a.  s.tudy  conducted  by  Lintz,  Loy,  Brock  & Potempa 
(1973) , problems  were  presented  to  users  with  alternative 
configurations  to  be  selected.  A procedure  somewhat 
similar  to  this  might  have  been  utilized  in  the  development 
of  the  assessment  problems.  For  obvious  reasons,  however, 
it  was  impossible  to  provide  the  variety  of  information 
supplied  by  Lintz  et  al  or  to  give  participants  the  same 
length  of  time  to  analyze  problems.  Nevertheless,  every 
care  was  taken  to  create  system  development  situations  which 
would  differentiate  between  decisions  made  using  the  HR 
handbook  and  those  made  without  using  it. 

To  facilitate  this  comparison  certain  problems  required 
use  of  the  prototype  handbook;  others  did  not;  decisions 
made  under  the  two  conditions  could  then  be  systematically 
compared.  Ideally  one  would  wish  to  be  able  to  say  for 
example  that  under  condition  A (no  HR  handbook)  26%  of 
decisions  were  correct,  whereas  under  condition  B (using 
the  HR  handbook)  76%  of  decisions  were  correct. 

2.  Prototype  handbook  utility  was  measured 
subjectively,  by  asking  participants  to  report  whether 
handbook  data  were  useful  in  solving  individual  problems; 
and  to  compare  the  usefulness  and  accessibility  of  their 
own  data  sources  with  the  prototype  handbook. 

3.  Prototype  handbook  acceptability  was  measured  by 
having  users  rate  the  usefulness  that  an  improved  HR 
handbook  would  have  in  their  work  and  the  degree  of 
influence  it  would  have  in  system  design. 

How  does  one  define  these  criteria  in  quantitative 
terms?  Is  50%  correct  solution  of  problems  adequate? 

75%?  85%?  Is  a rating  of  moderate  utility  for  the 

prototype  handbook  satisfactory,  or  does  one  require  a 
rating  :of  extremely  useful?  Obviously,  no  absolute 
quantitative  value  can  or  should  be  specified.  The 
quantitative  data  reported  in  Section  III  of  this  report 
must  be  considered,  however,  in  the  context  of  the 
engineer's  general  usage  of  behavioral  data  which,  even  for 
traditional  human  engineering  topics,  with  which  he  has  been 
familiar  for  a quarter-century,  is  not  great. 

QUESTIONNAIRE  FORMAT 

As  must  be  apparent  by  now,  it  was  determined  that  the 
assessment  format  should  take  the  form  of  a questionnaire 
containing  a series  of  problems  to  be  solved.  Certain  of 
the  problems  would  address  the  question  of  which  indexing 
system  was  preferable  (indexing  problems) ; others  would 
determine  how  well  participants  could  extract  data  from  the 
handbook  tables  (data  extraction  problems);  a special  set 
of  problems  would  compare  problem  solution  with  and  wi thout 
the  prototype  handbook. 
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Each  problem  to  be  solved  was  supplemented  by  questions 
(usually  in  the  form  of  a 5-point  Liker-type  scale)  asking 
the  participant  to  rate  factors  such  as  the  difficulty  he 
experienced  in  solving  the  problem,  his  estimate  of  the 
correctness  of  his  solution;  the  degree  to  which  he  had 
encountered  in  his  real-world  job  problems  similar  to  those 
presented  to  him. 

A final  section  of  the  questionnaire  asked  questions 
relating  to  the  prototype  handbook  as  a whole,  e.g.,  the 
user's  estimate  of  the  utility  of  the  handbook  as  a whole. 

The  questionnaire  will  be  described  in  detail  later. 
USER  PARTICIPANTS 

The  selection  of  personnel  for  this  assessment  was 
critical,  since  one  of  the  major  criteria  in  the  assessment 
was  the  acceptance  by  participants  of  the  need  for  the 
prototype  handbook.  As  indicated  in  Reed  et  al  (1975), 

"Any  handbook  development  problem  must  have  an  identified 
set  of  potential  users...  Engineers  of  different 
specialties,  system  planners,  etc.,  may  have  a need  for 
human  resources  data,  but  their  requirements  will  differ..." 
(p.  15) . Unfortunately,  "Many  specialists  feel  no  pressing 
need  for...  (HR)  ..  data,  or  are  totally  unaware  that  the 
data  exist"  (p.  16) . Participants  who  lacked  an  awareness 
of  HR  data  or  felt  no  need  for  it  might  provide  a 
misleading  picture  of  the  prototype  handbook's  utility.  In 
any  event,  the  target  audience  for  the  prototype  handbook 
was  limited  to  those  involved  in  the  design  and  development 
of  fire  control  subsystems. 

Early  in  the  planning  of  the  study  it  was  decided  that 
the  potential  participants  who  would  have  the  greatest  use 
for-  and  therefore  appreciation  cf  the  utility  of-  the 
prototype  handbook  would  be  those  that  were  responsible  for 
"high  level"  decisions  involving  tradeoffs  between  personnel 
and  other  engineering  factors. 

The  identification  of  specific  individuals  performing 
such  functions  is  very  difficult.  One  might  find  them  in 
Air  Force  system  development  and  headquarters  planning 
staffs,  but  such  personnel  are  not  readily  available  as 
study  participants.  One  might  also  find  them  in  companies 
responsible  for  development  of  major  sophisticated  systems. 

Since  the  prototype  handbook  was  developed  to  be  used 
by  both  Air  Force  and  contractor  personnel,  the  following 
strategy  was  adopted.  Two  samples  were  selected,  first, 
from  personnel  attached  to  System  Program  Offices  at  Wright- 
Patterson  AFB  (WPAFB) , the  second  from  engineers  working  on 
the  F-15  aircraft  development  at  McDonnell-Douglas  Aerospace 
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Company  (St.  Louis,  MO) . These  two  samples  permitted  a test 
of  the  question  whether  the  prototype  handbook  was  conveying 
information  of  the  same  effectiveness  and  utility  to  both 
sets  of  users.  Because  of  the  broadness  of  the  sample,  it 
would  probably  include  engineers  who  were  not  among  the 
target  user  audience.  The  purpose  of  the  broad  engineering 
spectrum  was  to  try  to  isolate  those  types  of  system 
development  personnel  who  would  find  the  prototype  handbook 
of  greatest  use. 

One  last  point,  it  is  necessary  to  emphasize  that  the 
problems  presented  were  to  assess  the  handbook,  not  to  test 
the  engineers'  ability  to  solve  the  problems. 

THE  QUESTIONNAIRE 

The  questionnaire,  as  finally  developed,  is  presented 
as  Appendix  A.  It  consists  of  12  problems  that  were 
developed  to  utilize  a representative  sample  of  the  data 
items  in  the  prototype  handbook.  Problems  were  extracted 
from  all  three  sections  of  the  handbook. 

1.  Handbook  Effectiveness.  Six  of  the  problems 
compared  the  Master  Index  (MI)  and  the  Alphabetical  Index 
(AI)  to  determine  which  indexing  system  was  more  effective. 
Problems  1 and  7 required  participants  to  use  the  MI  to 
determine  the  table  appropriate  to  the  solution  of  the 
problem  presented.  Problems  8 and  11  required  the 
participant  to  use  the  AI  to  do  the  same  thing.  Problems  3 
and  4 required  the  participant  to  choose  between  the  MI  and 
AI  in  finding  the  correct  table. 

In  the  indexing  problems  the  participant  was  merely 
required  to  determine  the  number  of  the  appropriate  table. 

In  indexing  problems  the  subject  did  not  have  to  solve  the 
problem. 

Problems  2,  5,  6,  9,  10  and  12  were  data  extraction 
problems.  These  were  identical  in  type  to  indexing 
problems  but  called  for  their  solution.  In  these  items  the 
participant  was  given  the  number  of  the  table  that  contained 
the  data  for  problem  solution.  Each  data  extraction 
problem  had  only  one  relevant  table  (the  one  specified  in 
the  problem) . No  cross-indexing  of  handbook  data  was 
required . 

Answers  to  all  problems  were  quantitative  and  could  be 
derived  directly  from  the  tabular  material  in  the  prototype 
handbook.  No  problems  were  presented  that  could  be  solved 
merely  by  expression  of  opinion;  e.g.,  design  configuration 
A is  better  than  (or  to  be  preferred  to)  B.  Such  problems 
were  avoided,  since  they  could  not  be  objectively  scored. 
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Problems  5 and  9 were  devised  with  a special  format  to 
permit  a comparison  of  solutions  "without"  and  "with"  the 
prototype  handbook.  Participants  were  asked  to  solve  these 
problems  first  without  reference  to  their  prototype 
handbook.  They  then  were  asked  (on  the  following  page  of 
the  form)  to  open  their  prototype  handbooks  to  the  specified 
table  and  resolve  the  problem,  either  to  confirm  their 
original  answers  or  to  find  a more  correct  one.  The 
correctness  with  which  the  problem  was  solved  under  the  two 
conditions  could  then  be  compared. 

(Why  did  participants  not  cheat  by  looking  immediately 
at  the  number  of  the  correct  table  and  solving  the  problem 
in  that  way?  Observation  suggests  that  they  did  not  do  so 
and  the  data  indicate  that  this  was  highly  unlikely,  since 
many  participants  failed  to  solve  the  problem  in  the 
"without"  condition.) 

Each  problem  had  of  course  only  one  correct  answer  as 
defined  by  the  relevant  handbook  table.  However,  partial 
credit  was  given  because  it  is  possible  to  enter  the 
prototype  handbook  at  varying  levels  of  efficiency.  Thus, 
in  extracting  data  from  a handbook  table  one  can  secure  an 
approximate  as  well  as  precise  answer.  For  example,  if  his 
answer  to  problem  1 (an  indexing  problem)  was  1.26-8.4,  he 
received  full  credit  (score  of  1.0).  If  he  recorded  1.26-8, 
he  received  half  credit  (0.5).  If  he  merely  identified  the 
correct  table  as  being  in  Section  I (or  recorded  no  answer 
at  all),  he  received  zero  credit  (0.0).  The  system  of 
partial  credit  was  adopted  for  all  12  problems. 

The  participant  was  also  asked  to  indicate  to  the 
nearest  minute  the  time  he  started  the  problems  and  when  he 
completed  it.  This  permitted  comparison  of  the  time 
required  by  each  of  the  various  conditions. 

Following  each  data  extraction  problem  (2,  5,  6,  9,  10 
and  12)  the  following  questions  were  asked: 

a.  How  easy  or  difficult  was  the  problem  to 

solve?  (This  measure  is  identified  in  Appendix  B as  Problem 
Difficulty,  q.  1.)  Problem  difficulty  is  an  obvious  • 

measure  with  which  to  differentiate  between  MI  and  AI , and 
between  the  "with"  and  "without"  conditions  of  problems  5 
and  9.  Moreover,  the  absolute  level  of  difficulty  the 
participant  experienced  is  one  way  of  evaluating  prototype 
handbook  utility;  if  perceived  difficulty  is  too  great, 
handbook  utility  is  reduced. 

b.  How  certain  are  you  about  the  correctness  of 
your  answer?  (Solution  Certainty,  q.  2.)  Personnel  might 
be  more  or  less  confident  about  their  decisions  depending 
on  the  conditions  they  were  exposed  to. 
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c.  In  your  work  have  you  encountered  problems 
similar  to  the  one  you  have  just  solved?  (Problem  Fidelity, 
q.  4.)  One  way  of  assessing  the  effectiveness  with  which 
engineers  can  use  the  prototype  handbook  is  in  terms  of  the 
representativeness  of  handbook  parameters. 

2.  Prototype  Handbook  Utility.  Questions  dealing  with 
handbook  utility  included: 

a.  If  in  your  work  you  encountered  a problem 
similar  to  the  one  you  have  just  solved,  would  the  material 
in  the  tables  have  been  useful  in  solving  that  problem? 

(HR  Data  Utility  q.  5.)  This  measure  of  handbook  utility  is 
for  the  immediate  problem  being  solved. 

b.  For  this  problem  how  useful  would  your  own 
data  sources  be  compared  to  the  prototype  handbook?  (Own 
Data  Utility,  q.  7.)  If  the  user  considers  his  own  data 
sources  as  useful  as  that  of  the  prototype  handbook,  he  is 
less  likely  to  use  that  handbook. 

c.  For  this  problem  how  accessible  would  your  own 
data  sources  be  compared  to  the  prototype  handbook?  (Own 
Data  Accessibility,  q.  8.)  Accessibility  of  own  data 
sources  is  another  way  of  assessing  handbook  utility.  If 
the  engineer's  own  data  sources  are  less  accessible  than 
the  prototype  handbook,  he  is  more  likely  to  use  the 
handbook . 


3.  Prototype  handbook  acceptability.  Questions 
dealing  with  handbook  acceptability  are  to  be  found  at  the 
end  of  Appendix  A and  are  also  listed  in  Appendix  B as  items 
A-D. 


a.  How  useful  would  an  improved  version  of  this 
type  of  handbook  be  in  the  work  you  do?  (Hdbk  Utility, 

Item  A,  Appendix  B.)  This  question  is  related  to  the  one 
on  HR  Data  Utility  (q.  5)  but  deals  with  the  prototype 
handbook  as  a whole. 

b.  How  influential  would  an  improved  handbook  of 
this  type  be  in  affecting  your  system  design  decisions? 
(Hdbk  Influence,  Item  B,  Appendix  B.) 

c.  How  often  have  you  used  human  resources  data 
in  your  work?  (Use  of  HR  data.  Item  C,  Appendix  B.) 

d.  How  important  do  you  consider  human  resources 
and  human  factors  in  operation  and  maintenance  of  avionics 
and  fire  control  equipment? 

e.  How  important  do  you  consider  human  resources 
data  in  affecting  system  design? 
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In  analyzing  the  resultant  data,  these  last  two 
questions  were  combined  under  the  heading  of  HR  Importance 
(Item  D,  Appendix  B.) 

4.  Prototype  Handbook  Improvement.  Participants  were 
also  asked  for  which  of  a number  of  subject  areas  could  such 
a prototype  handbook  be  used  effectively,  and  what  would 
they  recommend  to  improve  the  handbook? 

In  addition  to  the  above  questions,  the  participant  was 
given  a checklist  that  listed  any  difficulties  he  might  have 
experienced  in  solving  the  problem.  He  was  also  asked  to 
indicate  where  he  got  the  data  he  currently  used  to  solve 
such  problems . He  was  asked  to  comment  on  any  aspect  of  the 
problem  that  had  given  him  difficulty. 

For  indexing  problems  the  participant  was  asked  the 
following  questions: 

1.  How  easy  was  it  to  use  the  MI  (or  the  AI)  in 
finding  the  correct  table? 

2.  What  words  did  he  use  to  enter  the  index? 

3.  How  easy  was  it  to  perform  the  individual  steps  of 
the  MI  procedure,  or  (for  the  AI ) if  he  found  AI  difficult 
to  use,  he  could  check  one  of  several  possible  reasons? 

Following  the  series  of  12  problems,  the  participant 
was  asked  to  rate  the  usefulness  of  the  individual  sections 
of  the  handbook  tables;  did  the  tables  provide  as  much  data 
as  he  wished  and  if  not,  what  additional  data  did  he  wish? 

In  responding  to  the  above  questions  personnel  were 
given  a 5-point  Likert-type  rating  scale.  (It  was  felt  that 
users  would  be  confused  by  a more  complex  scale.)  For 
example,  the  ease/difficulty  dimension  had  as  major 
descriptors:  very  easy;  easy;  neither  easy  nor  difficult; 

difficult;  very  difficult.  The  certainty  dimension  ran 
from  completely  certain  (1.0)  through  very  certain  (2.5); 
neither  very  certain  nor  very  uncertain  (the  indifference 
point,  4.0);  very  uncertain  (5.5);  and  completely 
uncertain  (7.0).  In  all  cases  (except  those  dealing  with 
own  data  utility  and  accessibility)  the  positive  answer  to 
the  question  ranged  from  1.0  to  4.0.  Examination  of 
Appendix  A questions  will  indicate  the  various  descriptors 
that  were  used  for  the  individual  scales. 

MEASURES 

The  questionnaire  thus  permitted  the  following  measures 
which  the  reader  should  keep  in  mind  in  reading  Section  III 
of  this  report. 
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1.  Percentage  of  problems  correctly  solved  for 
categories  of  problems  and  types  of  users. 

2.  Mean  and  Standard  Deviation  (SD)  correctness  score 
for  individual  and  total  problems  and  mean  and  SD  for  a 
category  of  problems  (e.g.,  MI  vs  AI) . 

3.  Mean  and  SD  of  solution  times  (minutes)  for  the 
individual  problem,  for  a category  of  problems  and  for 
participants . 

4.  Mean  rating  and  SD  for  individual  questions,  for 
individual  problems  and  for  categories  of  problems  and 
participants . 

5.  Percentage  of  personnel  rating  questions  at  a 
specified  level  (e.g.,  4.0  or  less). 

6.  Percentage  of  personnel  choosing  MI  or  AI  on 
problems  3 and  4. 

7.  Frequency  of  checklist  items  checked,  as  in 
recommendations  for  improvement  (q.  7,  p.  31,  Appendix  A) . 

PARTICIPANT  BACKGROUND 

The  specialty  areas  of  engineers  selected  as 
participants  for  this  study  and  their  years  of  experience 
are  presented  in  Table  1.  Individual  user  background  data 
are  presented  in  Appendix  B. 

METHOD  OF  QUESTIONNAIRE  ADMINISTRATION 

Three  methods  of  questionnaire  administration  were 
possible : 


1.  One-on-one  interviews.  This  is  the  ideal  method 
and  was  used  by  Meister  & Farr  (1966)  because  it  permits 
one  to  explore  in  greater  detail  idiosyncratic  responses. 

The  drawback  is  the  inordinate  length  of  time  required  to 
secure  a reasonable  data  sample.  Under  the  conditions  of 
this  assessment,  interviews  were  not  practical. 

2.  Group  administration.  Participants  are  gathered 
together  in  small  groups  of  4 or  5 and  the  questionnaires 
are  administered  by  the  investigator.  The  advantage  here 

is  that  any  queries  can  be  answered  immediately,  participant 
performance  can  be  monitored,  and  the  group  administration 
creates  an  atmosphere  of  seriousness.  This  method  was  used 
with  the  largest  number  of  participants. 

3.  Ad  lib  administration.  Questionnaires  are  handed 
(or  mailed)  out  and  respondents  are  permitted  to  complete 
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TABLE  1.  PARTICIPANT  SPECIALTY  AREA  AND  EXPERIENCE 


Years  of 

No. 

Specialty  Area 

Experience 

WPAFB 

2 

Human  Engineering/Human  Factors 

(HE/HF) * 

10,  15 

2 

Maintainability  (Maint.) 

3,  4 

3 

Avionics  (Avion.) 

8,  18,  20 

1 

Aerospace  Ground  Equipment 
(AGE) 

4 

2 

Design  Engineer  (Design) 

(did  not 
answer) 

Total : 

10 

McDonnell- 

• 

Douqlas 

3 

Human  Engineering/Human  Factors 

(HE/HF) 

9,  18,  35 

3 

Maintainability  (Maint.) 

1,  10,  18 

2 

Training  (Train.) 

23,  32 

1 

Reliability  (Rel.) 

15 

2 

Crew  Systems  Design  (Crew) 

6,  21 

3 

Aerospace  Ground  Equipment 
(AGE) 

16,  18,  26 

1 

Avionics  (Avion) 

14 

1 

Operations  Analysis  (Ops.  An.) 

14 

7 

Design  Engineer  (Design) 

8,  10,  12,  25 
25,  25,  35 

3 

Miscellaneous  (Misc.) 

1,  5,  11 

Total : 

26 

*See  categories  in  Appendix  B. 
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the  forms  as  time  becomes  available  for  this  purpose.  This 
method  was  employed  with  a small  number  of  respondents  at 
WPAFB.  All  participants  had  an  opportunity  to  review  the 
prototype  handbook  before  they  completed  the  questionnaire. 
To  ensure  that  all  started  on  the  same  minimal  level,  a 
half  hour  indoctrination  preceded  the  administration  of  the 
questionnaire.  The  briefing  included  discussion  of  the 
purpose  of  the  assessment,  the  nature  of  HR  data,  the  two 
indexing  systems  and  the  nature  of  the  material  covered  by 
the  prototype  handbook.  All  user  questions  were  answered. 

The  questionnaire  was  then  completed  under  the 
investigator's  scrutiny  and  any  further  questions  answered. 
11  participants  were  identified  only  by  code  numbers.  They 
worked  at  their  own  speed  and  handed  in  their  forms  when 
completed.  In  general,  completing  the  questionnaire  took 
an  average  of  1.5  hours,  with  a few  respondents  working  an 
additional  half  hour.  The  investigator's  observations 
suggest  that  all  participants  were  unusually  assiduous  and 
serious  in  completing  their  questionnaires. 
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SECTION  III 


ANALYSES  AND  RESULTS 


INTRODUCTION 

The  data  on  which  the  analyses  in  this  section  are 
based  are  presented  in  Appendices  B and  C. 

Appendix  B lists  individual  participant  responses  to 
the  questions  in  the  assessment  questionnaire;  Appendix  C 
lists  the  same  data  summarized  across  participants  by 
problem. 

Before  proceeding  to  the  analyses  it  is  desirable  to 
examine  the  characteristics  of  the  data  because  these  bear 
on  the  statistics  employed.  In  order  to  make  use  of 
parametric  statistics  (e.g.,  Student's  t and  Pearson's  r) 
the  size  of  the  sample  should  be  around  25-30  and  the 
assumption  of  distribution  normality  (at  least)  must  be 
satisfied.  The  size  of  the  data  sample  in  this  study  is  36 
(assuming  that  data  from  WPAFB  and  McDonnell-Douglas  can  be 
combined — and  a test  of  this  point  showed  it  could  be 
combined) . However,  the  assumption  of  normality  of  distri- 
bution appeared  questionable  upon  inspection.  This  would 
appear  to  dictate  use  of  non-parametric  statistics,  although 
parametric  techniques  would  be  preferable  because  of  their 
ability  to  extract  the  full  value  of  the  available  data. 

Rather  than  lose  the  value  of  parametric  statistics, 
the  following  strategy  was  employed.  The  validity  of  each 
statistical  test  making  use  of  parametric  statistics  was 
checked  by  performing  a corresponding  appropriate  non- 
parametric  test.  If  the  non-parametric  test  results  did  not 
controvert  the  parametric,  the  parametric  test  results  were 
reported.  (All  non-parametric  tests  validated  parametric 
ones.)  On  the  ground  that  parametric  tests  exploit  the  data 
more  fully,  non-parametric  checks  were  not  made  for  any 
parametric  test  result  that  was  non-significant.  Thus,  if 
a t or  Pearson  r were  found  to  be  non-significant,  it  was 
assumed  that  a corresponding  appropriate  non-parametric 
test  would  also  indicate  non-significance.  (This  assumption 
was  checked  out  on  a sample  of  the  tests  made.) 

Section  III  has  been  organized  in  terms  of  the 
questions  this  study  sought  to  answer: 

1.  Can  engineers  use  the  handbook  to  solve  the  types 
of  HR  problems  for  which  the  handbook  was  developed? 

2.  Will  engineers  use  the  handbook? 
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3.  Which  types  of  engineers  (i.e.,  defined  by  their 
specialty  area)  are  most  likely  to  use  the  handbook? 

4.  Which  of  the  two  types  of  indexing  systems  do 
engineers  prefer  and  which  can  they  use  more  efficiently? 

5.  How  should  the  handbook  be  improved  to  render  it 
more  useful  and  acceptable? 

CAN  ENGINEERS  USE  THE  HANDBOOK 

Data  to  answer  this  question  come  from  two  sources: 

(a)  the  percent  of  the  12  problems  presented  that  engineers 
solved  correctly;  (b)  the  average  time  taken  to  solve 
these  problems. 

Percent  Correct 


Although  participants  had  little  or  no  opportunity  to 
study  the  handbook  before  using  it  to  solve  the  problems, 
they  solved  68%  of  the  problems  correctly.  Review  of 
Appendix  B indicates  very  great  variability  among  personnel; 
for  participants  1-10  from  WPAFB  the  range  was  25-92% 

(mean  64%);  for  11-36  from  McDonnell-Douglas  the  range  was 
slightly  more  constricted:  46-92%  (mean  70%) . 

Because  the  group  from  WPAFB  numbered  only  10,  the 
non-parametri c Mann-Whitney  U-test  (Siegel,  1956)  was  used 
to  compare  the  correctness  scores  of  the  two  groups  to 
determine  whether  these  could  be  combined.  The  difference 
between  the  two  groups  was  insignificant  (p  = .23). 
Consequently,  all  further  data  analyses  use  combined  data 
(N  = 36)  . 

One  may  ask  whether  a mean  of  68%  correct  means  that 
the  handbook  can  be  used  effectively.  In  view  of  the 
absence  of  prior  training  to  use  the  handbook,  and  the  fact 
that  most  respondents  were  somewhat  unfamiliar  with  the 
type  of  problems  presented  (see  subsequent  discussion  on 
Problem  Fidelity) , the  answer  to  that  question  appears  to 
be  yes . 

The  handbook  user  must  perform  two  essential  functions: 
(1)  to  determine  where  the  data  relevant  to  his  problem  are 
to  be  found  in  the  handbook;  (2)  to  extract  from  these 
data,  once  found,  those  principles,  guidelines  or  implica- 
tions that  will  enable  him  to  solve  the  problem  successfully. 

The  efficiency  of  the  first  function  could  be 
determined  by  studying  responses  to  the  indexing  problems 
(1,  3,  4,  7,  8,  11);  that  of  the  second  function,  by 
examining  responses  to  the  data  extraction  problems  (2,  5, 

6,  9,  10,  12).  If  one  type  of  problem  were  solved  more 
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effectively  than  the  other,  this  might  suggest  where 
emphasis  should  be  placed  in  improving  the  prototype 
handbook. 
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The  t test  (Guilford,  1965)  for  correlated  data 
was  used  to  test  the  difference  between  total  scores 
per  participant  for  indexing  problems  (mean  4.4,  SD  .94) 
and  data  extraction  problems  '(wean  3.5,  SD  .14).  The  t 
value  was  4.06  (35  df)  which  is  significant  at  better  than 
the  1%  level.  Participants  were  significantly  more 
efficient  in  finding  correct  tables  than  in  extracting  data 
from  them. 


The  difference  in  performance  between  the  two  types  of 
problems  may  have  resulted  in  part  from  problems  5 and  9 
which  were  considerably  more  difficult  for  participants 
than  the  other  two  data  extraction  problems. 

To  test  this  point  and  to  confirm  that  indexing 
problems  were  in  fact  more  efficiently  solved  than  data 
extraction  problems,  problems  5 and  9 were  eliminated  from 
the  data  sample  for  this  analysis  and  the  difference 
between  the  mean  correctness  percentage  of  indexing  (77%) 
and  data  extraction  problems  (67%)  was  tested,  using  t^  . 
This  test  confirmed  that  indexing  problems  were  in  ^ 

fact  solved  more  effectively  (t,  =2.32,  33  df,  signifi- 

cant at  the  5%  level) . 

Problems  5 and  9 were  specifically  developed  to 
determine  whether  the  "with"  handbook  condition  would 
produce  more  correct  solutions  than  the  "without"  condition. 
The  results  of  a t . test  between  the  two  conditions  are 
shown  in  Table  2.  ^ 


In  both  cases  performance  was  significantly  superior 
when  respondents  were  permitted  the  use  of  the  handbook. 
The  effect  was  substantially  greater  with  problem  9. 

TABLE  2.  CORRECT  SOLUTIONS  WITH  AND  WITHOUT  HANDBOOK 


Problem  5 Problem  9 


It  might  appear  obvious  that  if  participants  were 
permitted  use  of  the  handbook  to  solve  these  problems, 
solutions  would  in  fact  be  more  efficient.  However,  this 
depends  on  whether  the  necessary  information  can  be 
extracted  from  the  handbook.  A number  of  respondents  did 
not  improve  their  solutions  with  the  aid  of  the  prototype 
handbook.  It  is  possible  that  the  prototype  handbook  will 
be  most  useful  for  those  who  have  had  least  experience  in 
solving  HR  problems . 

Solution  Time 


Task  completion  or  solution  time  is  also  indicative  of 
efficiency  in  solving  problems.  On  the  average,  problem 
solution  time  was  around  5 minutes,  although  the  range  was 
much  broader  than  that  for  individual  problems.  Again,  no 
significant  differences  were  found  between  the  WPAFB  'group 
{mean  5.9  minutes,  SD  1.4)  and  that  from  McDonnell-Douglas 
(mean  5.7  minutes,  SD  1.6).  The  Mann-Whitney  U-test  for 
differences  in  mean  solution  time  resulted  in  a p of  .37, 
which  is  obviously  not  significant. 

Were  there  significant  differences  in  solution  times 
between  indexing  problems  (mean  6.0  minutes,  SD  2.0)  and 
data  extraction  problems  (mean  5.4  minutes,  SD  1.6)?  The 
tdep  value  is  1.96  (34  df)  which  just  misses  significance 
at  *the  5%  level.  The  slight  difference  in  completion  time 
favors  data  extraction  problems  over  indexing  problems . 

The  difference  suggests  that  indexing  tasks  may  have  been 
more  difficult  for  participants  than  data  extraction  tasks, 
a point  which  is  confirmed  in  the  next  sub-section. 

Problem  Difficulty 

Efficiency  in  solving  problems  must  be  considered  in 
terms  of  the  perceived  difficulty  of  the  problems  presented 
(see  Problem  Difficulty  9.1,  in  Appendices  B and  C) . This 
was  rated  on  a five-point  scale  ranging  from  very  easy  (1.0) 
to  very  difficult  (7.0).  Mean  difficulty  rating  for  the  12 
problems  was  3.6  with  a SD  of  1.5,  very  close  to  the  indif- 
ference point  (4.0).  Indexing  problems  (mean  3.7,  SD  0.9) 
appeared  to  be  significantly  more  difficult  (t^ep  = 5.62, 

34  df,  .01  level)  for  personnel  than  data  extraction 
problems  (mean  3.0,  SD  0.7).  This  suggests  that  partici- 
pants found  the  problem  of  discovering  the  correct  table 
number  to  be  greater  than  that  of  using  the  material  in  the 
table.  On  the  other  hand,  they  were  more  efficient  with 
indexing  problems  than  they  were  with  data  extraction 
problems.  Again,  there  are  wide  individual  differences. 

One  could  hypothesize  that  problems  solved  without  the 
use  of  the  prototype  handbook  (the  "without"  condition  of 
problems  5 and  9)  would  be  considered  by  participants  more 
difficult  than  the  same  problems  solved  with  the  handbook. 
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This  proved  to  be  the  case  for  problem  5 (t(jep=  2.04,  29  df, 
significant  at  the  2%  level),  but  not  for  problem  9.  The 
hypothesis  that  use  of  the  prototype  handbook  would  reduce 
the  apparent  difficulty  of  HR  problems  is  therefore 
partially  supported. 


Solution  Certainty 


Similar  results  are  found  for  participants'  feeling  of 
certainty  about  the  correctness  of  their  solutions  (see 
Appendix  C,  q.  2) . This  question  was  asked  only  of  -data 
extraction  problems.  The  mean  rating  of  certainty  was  3.7, 
with  a SD  of  0.8.  Again,  these  results  are  close  to  the 
indifference  point,  although  there  are  wide  individual 
differences  among  personnel  and  also  among  problems.  For 
example,  mean  solution  certainty  for  problem  10  was  2.7 
which  indicates  substantially  greater  certainty  than  that 
felt  for  other  problems,  particularly  problem  9. 

Certainty  is  important  in  comparing  the  with  and 
without  conditions  of  problems  5 and  9.  This  is  based  on 
the  assumption  that  one  of  the  effects  of  having  a handbook 
to  hedp  solve  HR  tradeoff  problems  would  be  to  increase 
personnel  confidence  in  their  decisions.  This  is  graphical- 
ly confirmed  in  Table  3 in  which  a lower  rating  indicates 
greater  confidence. 

* 

TABLE  3.  SOLUTION  CONFIDENCE  WITH  AND  WITHOUT  HANDBOOK 


Problem  5 Problem  9 


Measure 

Without 

With 

Without 

With 

Mean 

Certainty 

Rating 

5.4 

3.0 

5.3 

4.1 

SD 

1.5 

1.0 

1.4 

1.4 

fcdep 

7.83 

3.84 

N 

32 

32 

Sig.  Level 

.01 

.01 

WILL  ENGINEERS  USE  THE  HANDBOOK 

Even  if  data  are  valid,  if  engineers  will  not  use  these 
data,  their  utility  is  negated.  A number  of  measures  bear 
on  this  point.  These  are:  (1)  Problem  Fidelity  (q.  4); 

(2)  HR  Data  Utility  (q.  5);  (3)  Own  Data  Utility  (q.  7); 

(4)  Own  Data  Accessibility  (q.  8). 
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Problem  Fidelity 


Question  4 in  the  questionnaire  asked  whether  partici- 
pants had  encountered  problems  similar  to  the  one  they  had 
just  solved.  The  problems  in  the  questionnaire  were  derived 
directly  from  the  handbook  material  and  hence  indicate  the 
kinds  of  problems  for  which  the  handbook  will  be  useful.  If 
engineers  find  the  questionnaire  problems  different  from 
those  they  ordinarily  encounter,  they  may  not  find  the 
handbook  relevant  to  their  work.  The  question  of  problem 
fidelity  was  asked  only  of  data  extraction  problems  because 
only  for  these  were  personnel  required  to  utilize  tabular 
material . 

Mean  problem  fidelity  was  5.6,  SD  1.1.  On  the  scale  of 
1.0  (similar  problems  constantly  encountered)  to  7.0 
(similar  problems  never  encountered) , the  mean  rating 
suggests  that  respondents  had  some  difficulty  recognizing 
the  problems  presented  as  ones  they  ordinarily  encountered. 
Only  4 of  the  36  participants  were  positive  on  this  scale 
(at  or  below  the  neutral  point,  4.0).  The  engineer 
typically  encounters  his  problems  in  graphic  or  quantitative 
(e.g.,  tabular)  form.  Much  of  the  problem  unfamiliarity 
arose  because  the  questionnaire  problems  were  exclusively 
verbal . 

HR  Data  Utility 

Even  assuming  that  the  handbook  problems  were  unfamil- 
iar to  most  participants,  one  must  still  ask  whether,  if 
they  did  encounter  such  problems,  the  handbook  material 
would  have  been  useful.  Data  on  this  point  can  be  found  in 
Appendices  B and  C (HR  Data  Utility,  q.  5).  Responses  to 
this  question  are  more  positive,  the  mean  being  3.5,  SD  1.0. 
This  translates  to  a rating  of  moderately  to  very  useful. 

Own  Data  Utility 

The  utility  of  the  handbook  can  also  be  assessed  in 
terms  of  its  relationship  to  the  engineer's  own  data  sources. 
If  the  engineer's  own  data  (regardless  of  what  these  might 
be)  are  considered  by  him  to  be  as  useful  as  handbook  data, 
he  is  less  likely  to  make  use  of  the  handbook.  Again  the 
answer  is  somewhat  negative  (mean  4.7,  SD  1.2),  the  subjects 
considering  that  their  own  data  are  just  as  useful  or  only 
slightly  less  useful  than  handbook  material. 

The  responses  to  the  question  on  Own  Data  Utility 
probably  represent  a deep  conservatism  on  the  part  of 
engineers  who  prefer  their  own  familiar  data  sources  (no 
matter  how  deficient)  to  those  they  are  unfamiliar  with. 

This  is  a problem  that  the  handbook  developer  (and  the  human 
factors  engineer  also)  must  constantly  cope  with.  We  shall 
consider  this  problem  more  completely  in  the  Discussion 
section. 
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Own  Data  Accessibility 


On  the  other  hand,  many  engineers  feel  that  their  own 
data  sources  are  less  accessible  than  the  handbook  material. 
On  question  7,  Own  Data  Accessibility,  the  mean  rating  is 
5.2,  SD  0.9.  This  rating  is  significantly  different  from 
that  of  own  data  utility  as  shown  in  Table  4. 

TABLE  4.  COMPARISON  OF  OWN  DATA  UTILITY  AND 
ACCESSIBILITY  RATINGS 


Mean  Rating 
SD 

fcdep 

N 

Sig.  Level 


Own  Data  Utility  Own  Data  Accessibility 

4.7  5.2 

1.1  0.9 

4.30 
33 
.01 


The  greater  accessibility  of  handbook  material  compen- 
sates somewhat  for  the  engineer's  preference  for  his  own 
data . 


Handbook  Utility  as  a Whole 

The  previous  questions  dealt  with  the  utility  of  the 
handbook  as  it  related  to  specific  problems.  At  the  con- 
clusion of  the  evaluation  participants  were  asked  to 
indicate  how  they  felt  about  the  utility  of  the  handbook  as 
a whole,  assuming  it  had  been  improved  (Hdbk  Utility, 
item  A,  Appendix  B) . The  mean  rating  for  this  question  was 
4.8,  with  a SD  of  1.6.  This  corresponds  to  a feeling  that 
the  handbook  would  be  reasonably  useful. 


Assuming  that  a rating  of  4.0  or  less  on  the  scale 
would  indicate  an  attitude  on  the  part  of  participants  of 
positive  handbook  utility,  one  can  ask  what  percentage  of 
personnel  expressing  an  opinion  on  this  point  (two  did  not 
respond)  were  positive?  Fifteen  of  the  34  respondents  or 
44%  rated  handbook  utility  at  4.0  or  less.  Six  of  the  15 
or  18%  felt  that  the  handbook  would  be  very  or  extremely 
useful . The  44%  who  considered  handbook  utility  to  be 
moderately  to  extremely  useful  represented  nine  specialties 
(Human  Factors,  Maintainability,  AGE,  Operations  Analysis, 
Reliability,  Training,  Miscellaneous  and  Design).  It  is 
interesting  that  only  2 of  the  ’5  positive  respondents 
represented  general  design  engineering.  This  suggests  that 
those  who  found  the  handbook  most  useful  were  those  in 
highly  specialized  engineering  areas  that  are  more  likely  to 
require  tradeoff  analyses  involving  personnel.  One  might 
also  suppose  that  those  who  had  had  experience  with  similar 
problems  would  have  a more  positive  attitude  toward  the 
handbook  as  a whole.  A Pearson  r correlation  between 


ratings  of  Problem  Fidelity  and  Handbook  Utility  responses 
resulted  in  a correlation  of  +.50,  which  is  significant  at 
the  .01  level.  There  is  thus  a tendency,  as  one  might 
expect,  for  those  who  have  encountered  similar  problems  to 
feel  that  the  handbook  would  be  useful. 

Handbook  Influence 

Would  such  a handbook  be  influential  in  affecting 
system  design  (which  is  its  ultimate  purpose,  of  course)? 
Item  B,  Appendix  B provides  the  relevant  data.  The  mean 
rating  was  4.0  with  a SD  of  1.3.  This  represents  the  feel- 
ing that  such  a handbook  would  be  moderately  influential. 
Twenty-two  of  the  34  respondents  (62%)  rated  potential 
handbook  influence  as  4.0  or  less  (moderately  influential) . 
Ten  of  the  respondents  or  28%  felt  that  such  a handbook 
would  be  very  or  extremely  influential  (2.5  or  less  on  the 
scale) . The  same  engineering  specialties  that  responded 
positively  to  Handbook  Utility  also  responsed  positively  to 
this  question,  but  here  general  design  engineering  was  more 
heavily  represented. 

WHO  ARE  LIKELY  TO  BE  HANDBOOK  USERS 

Because  of  large  individual  differences  in  the  area  of 
engineering  specialization  and  years  of  experience,  one 
would  not  expect  every  engineer  to  make  use  of  the  handbook. 
Since  any  handbook  cannot  appeal  to  everyone,  one  would  wish 
to  determine  which  engineering  specialty  the  handbook 
appeals  to  most  or  with  which  it  is  most  effective? 

Since  the  handbook  dealt  with  HR  problems  and  its 
subject  matter  was  oriented  at  least  partially  to  mainte- 
nance themes,  it  would  be  interesting  to  contrast  the 
responses  of  Human  Factors  (HF)  and  Maintainability  (Maint.) 
engineers  with  those  of  other  types.  These  and  other 
comparisons  can  be  seen  in  Table  5.  In  essence,  all  differ- 
ences were  non-significant.  Consequently,  significance 
levels  have  not  been  reported. 

The  small  differences  in  performance  either  in  solving 
the  problems  or  in  attitude  toward  HR  data  and  handbook 
utility  among  participants  of  different  specialty  areas 
suggest  that  the  prototype  handbook  appeals  equally  to  all 
specialties . 

Each  of  the  rating  items  in  the  questionnaire  was 
developed  to  elicit  the  participant's  attitude  toward  the 
problems  and  the  prototype  handbook.  Examination  of  the 
interrelationships  among  these  ratings  might  therefore  help 
to  explain  individual  differences  in  problem  solution  and 
indicate  most  likely  users  of  the  prototype  handbook.  Using 
the  Pearson  r technique,  all  the  major  items  in  the 
questionnaire  (with  the  exception  of  solution  time  which  was 
considered*  to  have  only  secondary  significance)  were  inter- 
correlated.  The  results  can  be  seen  in  the  matrix  in 
Table  6. 
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TABLE  5a.  DIFFERENCES  IN  PARTICIPANT  PERFORMANCE 
AS  A FUNCTION  OF  SPECIALTY  AREA 


CORRECTNESS 


Measure 

HF/Maint 

Others 

Mean 

% 

61 

71 

SD 

17 

14 

Mean 

HF 

Maint. 

% 

60 

63 

SD 

13 

22 

Maint. 

Design  Eng/ 
AGE/A vi on 

Mean 

% 

63 

68 

SD 

22 

18 

Design  Eng/ 
AGE/Avion 

Others 

Mean 

% 

68 

69 

SD 

18 

13 

TIME 


Measure 

HF/Maint 

Others 

Mean 

Time 

5.0 

5.0 

(Min.) 

SD 

1.3 

1.6 

HF 

Maint. 

Mean 

Time 

4.9 

5.2 

(Min.) 

SD 

1.4 

0.6 

Design  Eng/ 

Maint. 

AGE/Avion 

Mean 

Time 

5.2 

5.6 

(Min.) 

SD 

0.6 

Design  Eng/ 

1.4 

AGE/Avion 

Others 

Mean 

Time 

5.6 

6.4 

(Min.) 

SD 

1.4 

1.4 

TABLE  5b.  DIFFERENCES  IN  PARTICIPANT  ATTITUDE 
AS  A FUNCTION  OF  SPECIALTY  AREA 


HR  DATA 

UTILITY 

OWN  DATA 

UTILITY 

Measure 

Mean 

HF/Maint 

Others 

Measure 

Mean 

HF/Maint 

Others 

Rating 

3.6 

3.4 

Rating 

4.4 

4.9 

SD 

1.0 

1.0 

SD 

0.8 

1.1 

HANDBOOK  UTILITY 
Measure  HF/Maint  Others 

Mean 

Rating  4.6  4.8 

SD  1.7  1.6 
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TABLE  6.  PEARSON  PRODUCT-MOMENT  CORRELATIONS  AMONG  QUESTIONNAIRE  ITEMS 


When  any  sizeable  number  of  correlations  are  run,  it 
would  be  expected  that  a certain  percentage  of  these  would 
prove  significant.  In  Table  6 r values  of  .30  or  higher 
(which  are  significant  at  the  .05  level)  should  be 
considered  only  as  indicative  and  possibly  worthy  of 
further  study. 

Certain  relationships  seem  logical.  For  example,  it 
has  already  been  mentioned  that  subjects  who  had 
encountered  similar  problems  might  well  think  better  of  the 
handbook  (Problem  Fidelity  - Handbook  Utility,  +.50). 

Those  who  felt  that  HR  data  had  utility  would  naturally  feel 
that  the  handbook  would  have  utility  and  be  influential  (HR 
Data  Utility-Handbook  Utility  and  Handbook  Influence,  +.40 
and  +.45).  Own  Data  Utility  and  Own  Data  Accessibility 
would  also  be  logically  related  (+.77).  Those  who  thought 
better  of  their  own  data  utility  than  of  the  handbook 
material  would  also  downgrade  Handbook  Influence  (-.35), 
although  the  relationship  between  Own  Data  Utility  and 
Handbook  Utility  was  exactly  zero.  Those  who  thought  their 
own  data  were  as  accessible  as  the  handbook  material  would 
probably  be  negative  to  Handbook  Influence  (-.56).  Those 
who  had  used  HR  data  in  the  past  (item  C,  Appendix  B)  would 
be  more  likely  to  consider  the  handbook  to  have  utility 
(+.34).  Those  who  felt  more  certain  about  their  solutions 
might  also  feel  more  positive  about  the  usefulness  of  HR 
data  (+.53).  Problem  difficulty  and  solution  certainty 
should  be  related  (+.55),  but  one  would  think  it  would  be  a 
negative  relationship;  probably  there  is  a more  important 
moderating  factor  that  is  not  known.  Problem  difficulty  is 
related  to  HR  data  utility,  but  the  relationship  is  not 
strong  (+.30).  Correct  solutions  were  related  to  a feeling 
of  own  data  utility  (+.41),  but  again  the  logic  of  the 
relationship  is  not  apparent. 

WHICH  INDEXING  SYSTEM  IS  PREFERABLE 

The  answer  to  this  question  requires  (1)  a comparison 
of  Master  Index  (MI)  responses  (problems  1,  7)  with 
Alphabetic  Index  (AI)  responses  (problems  8,  11)  in  terms  of 
correctness  and  completion  time;  (2)  analysis  of  the 
choices  made  by  respondents  when  they  had  the  opportunity 
to  select  between  the  two  systems  (problems  3 and  4) ; 

(3)  analysis  of  their  perception  of  the  difficulty  of  using 
the  two  systems. 

Statistics  for  correctness  and  time  data  are  shown  in 
Table  7. 
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It  is  obvious  without  the  necessity  of  doing  a formal 
statistical  test  that  responses  to  MI  problems  differ 
markedly  from  those  to  AI  problems.  The  question  is 
whether  this  difference  is  significant.  Because 
differences  within  MI  and  AI  problems  are  practically  non- 
existent (at  least  for  correctness) , responses  for  the  two 
MI  problems  were  combined  as  were  those  of  the  two  AI 
problems.  The  t<jep  value  for  combined  correctness  scores 
(1  & 7 vs  8 & 11)  was  5.18  (34  df ) , significant  at  the  1% 
level,  as  was  a comparable  test  solution  time  (t^ep  = 8.01, 
32  df) . Objective  performance  scores  therefore  favor  AI . 

Of  course,  the  problem  situations  with  which  the  two 
types  of  index  systems  were  compared  may  have  differed 
significantly  in  terms  of  difficulty  (apart  from  any 
difficulties  in  using  the  index) . This  turned  out  to  be 
the  case.  Perceived  difficulty  of  MI  problems  1 and  7 was 
2.9  and  5.0;  of  problems  8 and  11,  2.8  and  3.5.  When  the 
difficulty  ratings  given  problems  1 and  7 (ratings  of 
problem  difficulty  rather  than  difficulty  in  using  the 
index)  are  compared  with  those  of  problems  8 and  11,  t. 
is  3.26,  which  for  31  df  is  significant  at  the  1%  ™ 

level.  The  apparent  superiority  of  the  AI  system  must, 
therefore,  be  qualified  by  the  greater  easiness  of  the 
problems  in  which  alphabetic  indexing  was  required. 

In  two  problems  (3,  4)  participants  were  asked  to 
choose  one  of  the  two  indexing  schemes  (the  MI  or  the  AI 
system)  to  find  the  correct  table.  The  distribution  of 
choices  is  shown  in  Table  8 and  was  tested  using  the 
Chi-square  method  (Siegel,  1956)  . 

The  participant  was  given  four  alternatives.  He  could 
have  selected  (and  been  satisfied  with)  MI  or  AI,  or  he 
could  have  selected  MI  or  AI  first  and  then  have  switched 
to  the  alternative  indexing  system  if  the  first  choice 
proved  unsatisfactory.  If  those  choices  in  which  MI  was 
selected  first  but  then  rejected  (C)  were  added  to  the  MI 
choices  (A)  and  those  in  which  AI  was  selected  first  but 
then  rejected  (D)  were  added  to  the  AI  choices  (B) , the 
disparity  in  favor  of  MI  would  be  even  more  striking.  It 
is  apparent  that  engineers  preferred  the  MI  method, 
although  they  found  difficulties  in  using  it. 

Participants  were  also  asked  to  indicate  how  easy  or 
difficult  it  was  to  use  the  MI  and  AI  systems.  The 
distribution  of  difficulty  ratings  for  the  two  types  of 
index  systems  is  shown  in  Table  9. 
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Mean  Percent 
Correct 


61 


62 


94 


92 


fcdep  (1'  7 vs  8'  X1) 

= 3.12 

35 

.01 

.01 

45 

1 j 

’ >• 

Obviously,  use  of  the  MI  with  problem  7 and  AI  with 
problem  11  resulted  in  somewhat  greater  perceived 
difficulty.  Otherwise,  subjects  tended  to  view  both  MI 
and  AI  as  being  neither  particularly  easy  nor  difficult  to 
use.  To  determine  whether  MI  or  AI  presented  greater 
difficulty  to  subjects,  ratings  of  1 and  7 combined  and  8 
and  11  combined  were  compared,  using  t^  . This  resulted 

in  a value  of  3.12  which  was  significant  at  the  1%  level. 
Quite  apart  from  the  difficulty  of  the  problems  to  which 
they  are  applied,  the  indexing  methods  are  differentially 
difficult,  the  MI  being  perceived  as  significantly  more 
difficult  than  the  AI,  even  though  respondents  preferred 
to  use  MI  when  they  had  the  opportunity.  Of  course,  it  is 
not  completely  possible  to  differentiate  the  inherent 
difficulty  of  an  indexing  system  from  the  difficulty  of  the 
problem;  a more  difficult  problem  will  demand  more  of  an 
indexing  system. 

The  difficulty  involved  in  applying  each  of  the  three 
major  MI  steps  was  also  explored.  In  each  of  the  two  MI 
problems  (1,  7)  subjects  were  asked  to  rate  the  difficulty 
of  each  step.  The  results  are  shown  in  Table  10. 


TABLE  10.  PERCEIVED  DIFFICULTY  OF  MI  STEPS 

Problem  1 Problem  7 

Steps  ABC  ABC 

Mean  Rating  2.8  2.9  3.0  4.5  4.6  4.2 

SD  1.5  1.5  1.5  1.8  1.9  1.9 

Step  A - Determine  section  number  from  Master  Index. 

B - Find  the  appropriate  index  number  from  the 
Master  Index  Table  of  Contents. 

C - Determine  the  table  number  from  the  Master 
Index  Table  of  Contents. 


The  differences  between  steps  within  problems  1 and  7 
are  obviously  miniscule,  suggesting  that  all  steps  are 
equally  easy  or  difficult.  In  problem  1 the  ratings 
cluster  between  "easy"  (2.5)  and  "neither  easy  nor 
difficult"  (4.0).  In  problem  7,  the  ratings  are  on  the 
difficult  end  of  the  continuum.  Obviously  there  is  a 
relationship  between  the  difficulty  of  the  problem  and  the 
difficulty  in  applying  the  index. 
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RECOMMENDATIONS  FOR  IMPROVEMENT 

The  recommendations  to  be  described  derive  in  many 
cases  from  the  comments  participants  made.  Question  3 in 
each  data  extraction  problem  provided  a list  of  items  (see 
Table  11)  which  personnel  could  check  if  they  had  difficulty 
with  the  problem.  With  2988  possible  items  to  be  checked 
(36  participants  x 14  items  x 6 problems  - 36)*,  a total  of 
158  responses  or  19%  were  made.  Most  comments  were  made  in 
relation  to  problems  5 and  9,  because  these  were  perceived 
as  most  difficult.  More  important  is  the  nature  of  the 
items  checked.  Item  2,  not  the  kind  of  problem  I am 
familiar  with,  was  checked  most  often.  The  emphasis  on 
problem  familiarity  is  something  we  have  encountered  before. 
However,  only  5 responses  were  made  to  item  5,  problem  not 
realistic.  Apparently  participants  accepted  the  problems 
as  realistic.  (This  point  is  important  as  tending  to 
confirm  the  validity  of  the  problem  situations  developed 
for 'the  questionnaire.)  Respondents  felt  that  in  some 
cases  not  enough  data  were  available  to  solve  the  problem 
(item  4)  and  that  the  Implications  of  the  data  were  not 
clearly  pointed  out  (item  12) . This  is  a common  complaint 
encountered  by  many  human  factors  engineers  working  in 
system  development.  There  were  some  comments  also  on  the 
difficulty  of  extracting  data  from  handbook  tables  (item  7) 
and  reluctance  to  use  the  handbook  to  solve  the  problems 
(item  8) . 

Tables  12  and  13  present  recommendations  made  by 
participants  for  ways  of  using  and  improving  the  handbook. 
(Frequencies  for  individual  items  do  not  add  up  to  36 
because  many  respondents  failed  to  answer  this  question.) 
They  saw  a wide  range  of  uses  for  the  handbook,  as 
evidenced  by  the  distribution  of  comments.  The  handbook 
could  be  improved  by  making  data  more  current  and  the 
indexing  system  easier  to  use.  Observations  of  participant 
behavior  (while  they  were  completing  the  questionnaire) 
suggest  that  handbook  acceptance  would  have  been  increased 
had  both  indexing  systems  been  more  efficient. 

The  following  recommendations  for  improvement  of  the 
prototype  handbook  can  be  made  on  the  basis  of  participant 
comments  and  observations  by  the  investigator: 

1.  The  categories  in  the  MI  (see  Figure  3)  appear  to 
be  too  gross  and  should  be  made  more  detailed  (perhaps  by 
increasing  the  number  of  categories) . The  data  parameters 
which  engineers  derive  from  their  problems  and  with  which 


*One  item  was  inadvertently  left  out  of  problem  2. 
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TABLE  11.  DISTRIBUTION  OF  PARTICIPANT 
COMMENTS  ON  PROBLEMS 


Didn't  understand  211500  9 

problem. 

Not  the  kind  of  problem  893612  29 

I am  familiar  with. 

Did  not  have  enough  time  010000  1 

to  solve. 

Not  enough  data  to  solve  3 1 4 13  0 3 24 

problem . 

Problem  not  realistic.  -*01310  5 


Data  irrelevant  to  00 

problem. 

Data  difficult  to  extract  1 4 

from  table. 

Can't  solve  problem  0 13 

without  handbook. 

Information  in  table  too  1 3 

verbal . 

Data  not  current.  1 1 


Organization  of  table 
confusing . 

Implications  of  data  not 
clearly  pointed  out. 

Too  much  information  in 
table . 

Too  much  irrelevant 
information. 

Total : 
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TABLE  12.  DISTRIBUTION  OF  PARTICIPANT  RECOMMENDATIONS 
FOR  HANDBOOK  USE 


Use  Frequency 


Avionics  and  fire  control  system  historical  data  10 

Acquisition  cost  of  avionics  and  fire  control 

systems  7 

Equipment  design  trade-off  19 

Description  of  avionics/fire  control  tasks/ jobs  8 

Human  error  prediction  14 

Manpower  costs  15 

Personnel  selection  criteria  7 

Manpower  skill  and/or  availability  11 

Training  costs  and  training  time  18 

Logistics  costs  and  trade-off  15 

Maintainability  and  maintenance  performance  times  20 

Manpower  life  cycle  costs  8 

Prediction  of  technician  performance  8 

Fire  control  design  characteristics  6 


TABLE  13.  DISTRIBUTION  OF  PARTICIPANT  RECOMMENDATIONS 
FOR  HANDBOOK  IMPROVEMENT 


Recommendation  Frequency 

Improve  overall  handbook  organization  5 
Improve  organization  of  individual  tables  6 
Make  data  more  current  21 
Increase  amount  of  data  in  tables  9 
Provide  more  data  points  8 
Reduce  verbiage  in  tables  3 
Present  data  in  more  quantitative  form  6 
Make  index  easier  to  use  19 
Cover  more  subject  areas  12 
Reduce  amount  of  material  in  tables  0 
Indicate  uses  of  data  more  effectively  8 
Broaden  data  sources  6 
Provide  more  graphic  illustrations  8 
Make  instructions  for  indexing  system  clearer  11 
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they  approach  the  MI  are  usually  quite  detailed  and  taken 
directly  from  the  problem  as  they  conceptualize  it;  e.g., 
"grade  levels"  (problem  5) . If  the  term  "grade  level"  or 
some  category  fairly  close  to  it  is  not  available  in  the 
MI,  the  engineer  has  difficulty  finding  the  correct  term, 
because  now  he  must  search  for  some  more  molar  term  which 
encompasses  "grade  level."  The  same  comment  might  be  made 
about  the  AI : the  number  of  detailed  categories,  particu- 

larly those  emphasizing  interrelationships,  should  be  much 
expanded. 


2.  The  Implications  section  of  the  handbook  tables  is 
in  most  cases  fairly  abstract  and  non-communicative . To 
the  engineer  the  term  "implications"  suggests  that  the 
category  will  explain  the  conclusions  that  should  be  drawn 
from  the  tabular  or  graphic  material  on  the  page.  When  the 
category  does  not  provide  this  explanation,  the  engineer 
feelp  frustrated.  It  may  be  that  the  prototype  handbook 
developers  were  overly  conservative  in  what  they  felt  they 
should  say  about  their  data.  The  Implications  section  of 
each  table  should  point  out  the  specific  applications  of 
the  data  and  the  conclusions  that  seem  warranted  from  these 
data.  Presently  the  Implications  section  is  too  narrowly 
related  to  fire  control  subsystem  data.  If  the  prototype 
handbook  developers  want  to  make  the  handbook  more 
generally  usable,  generalizations  that  extend  beyond  the 
fire  control  subsystem  data  on  which  Implications  is  based 
should  be  indicated. 

3.  The  section  of  the  data  tables  dealing  with  Models 
for  Data  Application  should  be  eliminated,  at  least  until 
these  models  can  in  fact  be  applied  to  the  data. 

4.  It  has  already  been  mentioned  that  engineers 
prefer  their  data  to  be  as  recent  as  possible.  Limitations 
on  data  availability  may  make  this  difficult  to  accomplish, 
but  the  effort  should  be  attempted.  (See  later  comments  on 
computeri zation . ) 

5.  The  explanation  of  the  way  in  which  the  indexing 
systems  should  be  used  (which  is  included  in  the  prototype 
handbook  Introduction)  and  the  way  in  which  data  should  be 
extracted  from  tables  (which  is  not  fully  explored  ih  that 
Introduction)  should  be  considerably  amplified.  It  would 
be  desirable  to  expand  the  handbook  Introduction  to  include 
a number  of  sample  problems  which  the  user  would  work 
through  to  a specified  correct  answer,  preferably  in  a 
programmed  learning  manner. 

6.  The  kinds  of  problems  for  which  the  prototype 
handbook  is  most  appropriate  should  be  clearly  spelled  out, 
with  illustrative  situations.  Some  of  the  participants' 
initial  uneasiness  in  using  the  handbook  undoubtedly 
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IV.  DISCUSSION 


This  study  began  with  a number  of  major  questions: 

Can  engineers  use  the  handbook?;  Will  they  use  it?;  Which 
of  the  various  types  of  engineers  are  most  likely  to  use 
the  handbook?  At  the  conclusion  of  the  study  the  first  two 
questions  can  be  answered  positively;  as  for  the  third 
question,  the  handbook  appeals  to  all  engineering 
specialties . 

With  regard  to  the  question,  "Can  engineers  use  the 
handbook?"  the  fact  that  without  substantial  training  to 
utilize  the  indexing  systems  and  the  tabular  material,  68% 
of  the  problems  presented  were  solved  correctly,  either 
wholly  or  partially,  satisfies  the  criterion  of  effective- 
ness. Had  participants  been  indoctrinated  more  fully  on 
how  to  use  the  handbook  to  solve  problems,  this 
effectiveness  measure  would  have  been  significantly  higher. 
Moreover,  participants  performed  significantly  better  on 
problems  5 and  9 with  the  handbook  than  without  it  and 
considered  solution  of  problem  5 without  the  handbook  to  be 
significantly  more  difficult  than  solution  of  the  same 
problem  with  the  handbook.  Their  confidence  in  handbook 
solutions  was  rated  significantly  greater  than  their 
confidence  in  solutions  without  the  handbook. 

Objectively,  therefore,  what  one  can  term  the 
efficiency  of  the  handbook,  as  defined  by  correct  solutions, 
and  its  utility , as  defined  by  participants'  increased 
decision  confidence  and  reduced  problem  difficulty,  both 
favor  the  handbook. 

But  will  engineers  voluntarily  use  the  handbook?  This 
has  always  been  the  difficulty  in  the  transmission  of 
behavioral  inputs  to  the  engineer. 

The  answer  to  this  question  is  reasonably  positive. 

If  engineers  feel  that  a product  will  be  useful, they  will 
use  it.  Respondents  felt  that  the  handbook  would  be 
moderately  to  very  useful  in  solving  problems  of  the  type 
the  handbook  did  purport  to  solve.  Forty-four  percent  of 
the  participants  felt  that  the  handbook  had  positive 
utility  and  62%  felt  that  the  handbook  would  influence 
their  design.  Although  they  felt  that  their  own  data 
sources  were  almost  as  good  as  those  of  the  handbook,  the 
former  were  less  accessible. 

With  regard  to  the  question,  "Which  of  the  various 
types  of  engineers  represented  in  the  sample  of  participants 
are  most  likely  to  use  the  prototype  handbook?"  the  data 
are  merely  suggestive.  Attempts  to  compare  the  performance 
of  engineers  in  terms  of  their  specialty  areas  elicited  no 
significant  differences  which  suggests  that  the  handbook 
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appeals  to  all  engineering  types.  Based  on  the  ratings  of 
handbook  utility,  there  is  some  suggestion  that  engineers 
who  perform  in  more  specialized  design  efforts  (e.g.. 
Reliability,  Maintainability,  Training,  etc.),  are  more 
likely  to  be  positive  toward  handbook  use  then  engineers 
in  general  design.  This  ties  in  also  with  the  hypothesis 
(the  most  one  can  call  it  because  it  is  based  on  a 
correlation  of  only  +.50)  that  engineers  who  had 
encountered  similar  problems  previously  tended  to  be  more 
positive  toward  the  handbook  than  others. 

The  failure  to  discover  distinctive  user  types  in  this 
evaluation  may  also  mean  that  the  target  audience  for  this 
handbook  is  not  to  be  found  in  the  engineer's  specialty 
area  as  much  as  it  is  in  the  type  of  job  he  does,  a type  of 
job  that  requires  solution  of  problems  for  which  handbook 
material  is  appropriate.  It  may  be  that  the  kinds  of 
problems  for  which  the  prototype  handbook  is  most  adaptable 
are  found  mostly  in  analytic,  planning  and  managerial 
positions  within  the  Air  Force  and  contractor  facilities. 

The  question  of  whether  it  is  worthwhile  on  a life-cycle- 
personnel-cost  basis  to  automate  a new  system  is  the  sort 
of  question  asked  very  early  in  system  development,  at  the 
stage  at  which  analyses  and  planning  must  be  performed,  as 
in  developing  the  Air  Force's  Operational  Requirement  or 
Required  Operational  Capability.  Lower  level  system 
development  personnel  for  whom  behavioral  data  handbooks 
are  usually  written  do  not  often  deal  with  such  problems. 

If  one  were  to  sum  up  what  this  assessment  reflects, 
it  might  be  stated  somewhat  as  follows:  The  prototype 

handbook  is  a useful  tool  and  personnel  using  it  can  solve 
certain  kinds  of  problems  more  effectively  than  they  could 
otherwise.  Because  of  this,  it  should  be  improved  by 
updating  its  data  and  sharpening  their  implications,  by 
simplifying  the  MI  system  and  reducing  unnecessary  verbiage. 
To  increase  its  utilization  the  introduction  of  the 
improved  handbook  should  be  preceded  by  a deliberate  user 
indoctrination  effort.  In  addition,  a special  effort 
should  be  made  to  identify  the  most  promising  target 
audience  for  this  material.  It  must  be  recognized,  as 
Reed  et  al.  (1975)  did,  that  all  system  development 
personnel  do  not  have  the  same  needs  for  behavioral  data. 
Pinpointing  the  most  receptive  audience  for  the  prototype 
handbook  will  increase  its  usefulness. 

HR  scientists  who  develop  such  handbooks  must 
ultimately  cope  with  the  philosophical  questions  with  which 
this  report  began: 

1.  Should  HR  data  handbooks  be  geared  for  use 
directly  by  system  development  personnel  without  (or  with 
only  minimal  assistance  by)  the  behavioral  specialist?  Or 
should  such  handbooks  be  directed  primarily  at  behavioral 
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specialists  to  assist  them  in  making  HR  inputs  to 
developers?  The  choice  between  these  alternatives  depends 
on  whether  one  thinks  the  system  developer  has  the 
necessary  behavioral  background  to  "go  it  on  his  own."  It 
depends  on  whether  the  handbook  developer  can,  on  his  own 
(i.e.,  without  the  participation  of  the  human  factors 
engineer  in  the  development  context) , translate  behavioral 
data  into  engineering  equivalents  that  will  satisfy  his 
development  needs. 

The  answers  to  the  above  questions  have  implications 
for  the  type  of  handbook  developed  and  methods  of  assessing 
its  usefulness.  It  is  related  also  to  the  role  one 
conceptualizes  for  the  human  factors  engineer  working  in 
system  development:  whether  he  is  to  be  the  primary 

initiator  of  HR  inputs  to  the  engineer  (in  which  case 
handbook  material  ought  to  be  more  directly  aimed  at  him) 
or  mqre  passive  relative  to  the  engineer. 

The  answers  to  these  questions  are  not  of  course 
completely  dichotomous,  because  emphasizing  one  approach 
does  not  require  rejecting  the  other  completely.  Whatever 
the  answer,  however,  consideration  of  these  questions  is 
required  because  it  is  likely  that  many  HR  data  handbooks 
are  developed  in  ignorance  of  the  real  contexts  in  which 
these  handbooks  will  be  used — or  not  used.  For  this  reason 
research  into  handbook  methodology — the  kinds  of  problems 
faced  by  system  development  personnel  and  the  data  they 
need,  the  ways  they  have  of  using  behavioral  inputs — is  a 
necessary  precursor  of  effective  handbook  development. 

2.  Should  the  handbook  developer  conceptualize  the 

target  audience  for  HR  data  handbooks  as  very  broad  (e.g. 
specialists  such  as  training  personnel,  human  factors, 
reliability  and  maintainability  engineers  and  general 
design  engineers)  or  should  one  attempt  to  find  a more 
restricted  specialist  audience?  The  broader  the  audience 
the  more  difficult  it  is  to  satisfy  any  segment  of  that 
audience.  A very  "narrow"  audience  may  be  thought, 
however,  to  lack  cost-effectiveness;  some  may  feel  that 
handbooks  are  profitable  only  when  they  have  broad  appeal. 
But  is  it  possible  to  appeal  to  a mass  audience  if  , 

differences  among  engineers  in  background  and  what  ^hey  do 
are  so  large? 

3.  Should  one  approach  the  compilation  of  behavioral 
data  by  attempting  to  find  whatever  data  are  available  and 
assembling  them;  or  should  one  start  first  by  determining 
the  kinds  of  problems  for  which  data  are  needed  and  then 
assembling  only  data  relevant  to  selected  problems?  The 
general  scarcity  of  behavioral  data  has  probably  led  to  a 
concentration  on  the  first  approach,  although  it  would  be 
wrong  to  assume  that  handbook  developers  have  wholly 
ignored  the  problems  for  which  data  are  needed. 


Despite  the  difficulty  of  the  questions  posed  above, 
it  is  obvious  that  there  will  be  continuing  attempts  to 
develop  behavioral  data  handbooks.  And  logically  so,  since 
not  even  an  "expert"  in  the  behavioral  sciences  has  all  the 
information  he  needs  when  he  needs  it. 

Human  factors  research  is  performed  in  order  to 
provide  data  that  will  help  solve  difficult  system 
development  problems.  In  order  to  make  that  research 
meaningful,  its  results  must  be  communicated  to  a user, 
whoever  that  user  is . Communication  to  anyone  other  than 
the  very  highly  specialized  researchers  who  perform  this 
research  demands  a vehicle,  which  is  the  handbook.  Indeed, 
some  human  factors  engineers  feel  that  there  is  no  economic 
justification  to  support  human  factors  research  for  which 
results  cannot  be  communicated  in  handbook  form. 

From  that  standpoint,  one  of  the  major  functions  the 
behavioral  scientist  must  perform  is  the  communication  of 
his  research  findings  in  the  form  of  handbooks.  Technical 
reports  and  scientific  journals  do  not  satisfy  the 
informational  needs  of  system  developers,  because  they  have 
no  time  to  find  them  and  even  specialists  have  difficulty 
assembling  these  materials. 

In  this  connection  consideration  should  be  given  to 
the  use  of  computerized  information-retrieval  systems  for 
the  provision  of  HR  data.  Since  HR  data  tend  to  decay 
rapidly,  computer-based  systems  may  be  more  efficient  in 
updating  data  than  hardcopy  volumes.  The  cost  involved  for 
the  former,  while  high,  is  not  significantly  greater  than 
that  of  compiling  data  for  handbooks,  since  the  underlying 
preparatory  work  is  the  same  in  either  case.  The  same 
factors  that  apply  to  handbook  development  must  also  be 
considered  in  computerized  data  systems. 

Since  technical  reports  and  scientific  journals  are 
concerned  largely  with  research  and  not  the  application  of 
research  findings,  the  handbook  or  its  computerized 
equivalent  must  be  the  means  of  translating  behavioral  data 
into  system  development  terms.  Even  more  than  communication, 
the  translation  function  provides  an  even  more  important 
rationale  for  the  handbook. 

For  HR  data,  which  are  much  more  difficult  to  handle 
than  more  molecular  human  engineering  data,  the  handbook  is 
even  more  a necessity.  The  prototype  handbook  whose 
assessment  is  discussed  in  this  report  is  an  excellent  start 
in  that  direction. 
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V.  CONCLUSIONS  AND  RECOMMENDATIONS 


The  following  conclusions  are  derived: 

1.  System  development  personnel  can  utilize  the 
prototype  handbook  to  make  significantly  more  correct 
decisions  than  without  the  handbook.  This  is  evidenced  by 
the  fact  that  they  solved  68%  of  the  problems  presented 
and  performed  significantly  more  effectively  with  the 
handbook  as  an  aid  than  without  it.  With  the  handbook, 
engineers  had  greater  confidence  in  their  decisions  and 
felt  that  the  problems  were  less  difficult  to  solve. 

2.  Based  on  the  assumption  that  those  who  see  utility 
in  a product  will  use  it,  it  can  be  expected  that  a very 
substantial  number  of  engineers  will  use  the  handbook. 
Forty-four  percent  of  participants  saw  the  handbook  as 
having  utility;  62%  believed  it  would  influence  design. 
Moreover,  although  they  considered  their  own  data  sources 
as  good  as  the  handbook,  the  former  were  much  less 
accessible . 

3.  Those  who  see  utility  in  the  handbook  are  more 
likely  to  have  specialized  jobs  (e.g..  Human  Factors, 
Maintainability,  Crew  Station  Design)  than  general  design 
functions.  Based  on  a correlation  of  +.50  between  Problem 
Fidelity  and  Handbook  Utility,  it  appears  that  those  who 
have  used  handbooks  to  solve  trade-off  problems  are  more 
positive  to  the  handbook. 

4.  Engineers  preferred  to  use  the  Master  Index 
system  rather  than  the  Alphabetical  Index,  but  experienced 
greater  difficulty  with  the  former.  They  performed  more 
effectively  with  the  Alphabetical  Index,  presumably  because 
they  were  more  familiar  with  it. 

5.  A number  of  improve'ments  were  recommended  by 
participants,  including  updating  data,  simplifying  the 
Master  Index  system,  reducing  verbiage  in  the  tables  and 
clarifying  the  implications  of  handbook  data. 

The  following  recommendations  are  made: 

1.  The  results  of  this  assessment  study  are  suffi- 
ciently encouraging  to  warrant  continuing  the  effort  to 
develop  HR  data  handbooks. 

2.  The  development  of  a full-scale  handbook  should 
consider  the  following: 

a.  The  MI  system  should  be  revised  to  expand  its 
categories  and  to  make  them  more  detailed. 


b.  The  Implications  section  of  the  data  tables 
should  be  written  in  more  detailed  fashion,  explicitly 
specifying  the  design  conclusions  that  may  be  derived  from 
the  data. 


c.  The  Introduction  should  be  considerably 
expanded  to  make  it  an  aid  in  teaching  the  use  of  the 
handbook.  The  Introduction  should  include  a listing  of 
the  types  of  system  development  problems  for  which  the 
handbook  is  most  useful  and  a number  of  sample  problems 
that  can  be  worked  through  by  the  user  on  a step-by-step 
basis  to  a specified  answer. 

d.  The  data  contents  should  be  updated  as  much 
as  possible. 

e.  The  material  in  Section  III  of  the  prototype 
handbook  that  deals  with  parameters  that  cut  across  the 
individual  fire  control  subsystems  and  apply  to  systems  in 
general  should  be  expanded.  This  will  increase  the  utility 
of  the  handbook. 

3.  The  assessment  presented  in  this  report  was  limited 
to  the  concept  and  structure  of  a HR  handbook.  Assessment 
of  a full-scale  handbook  should  be  conducted  during  the 
development  of  the  handbook  and  should  emphasize  the  validity 
of  the  data  contents. 

4.  In  line  with  concepts  described  in  the  Discussion 
section,  consideration  should  be  given  to  supporting 
research  designed  to  discover  the  most  effective  target 
audience  for  HR  material.  The  handbook  development  process 
should  be  considered  as  much  more  than  mere  compilation  of 
data.  This  requires  research  to  determine  the  target 
audience  before  the  handbook  is  developed  and  determination 
of  that  audience's  needs  and  characteristics.  Consideration 
should  also  be  given  to  the  development  of  HR  handbooks  that 
deal  with  operator/maintainer  behavior  in  relation  to 
generalized  equipment  parameters  that  cut  across  specific 
subsystems . 

5.  A study  should  be  performed  to  investigate  the 
feasibility  of  using  computerized  information-retrieval 
techniques  to  substitute  for  hardcopy  handbooks.  A hybrid 
system  involving  the  use  of  a computer  to  prepare 
periodically  updated  hardcopy  handbooks  should  be 
investigated. 
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appendix  a 

THE  PROTOTYPE  HANDBOOK  QUESTIONNAIRE 

Problem  1 Start  Time  

In  field  shop  repair  of  fire  control  systems,  what  is  the 
relationship  between  occupational  job/duty  and  performing  skill 
level?  Using  the  Master  Index,  find  the  number  of  the  table 
describing  this  relationship. 

ANSWER:  Table  

In  the  questions  below  check  anywhere  along  the  scale. 


1.  how  easy  or  difficult  was  the  problem  to  solve? 


r~ 

Very 

Easy 


Easy 


1 1 1 

Neither  Easy  Difficult  ,Velfy 

Nor  Difficult  Difficult 


2.  How  easy  was  it  to  use  the  Master  Index  in  finding  the 
correct  table? 


I — 
Very 
Easy 


Easy 


Neither  Easy Difficult 
Nor  Difficult 


1 

Very 

Difficult 


3.  With  what  word  or  words  did  you  enter  the  Index  first? 
If  you  used  additional  terms,  what  were  they? 


4.  How  easy  was  it  to  perform  the  individual  steps  of  the  Master 
Index  procedure? 

A.  To  determine  section  number  from  Master  Index: 


Very 

Easy 


— r~ 

Easy 


1 ] I 

Neither  Easy  Difficult  Very 

Nor  Difficult  Difficult 


B.  To  find  the  appropriate  index  number  from  the  Master 
Index  Table  of  Contents? 


I — 

Very 

Easy 


— I — 
Easy 


Neither  Easy  Difficult 
Nor  Difficult 


Very 

Difficult 
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I . 


[ 1 

! f 


1 

I 


C.  To  determine  the  table  number  from  the  Master  Index 
Table  of  Contents? 


I — 
Very 
Easy 


— I — 
Easy 


1 r 

Neither  Easy  Difficult 
Nor  Difficult 


1 

Very 

Difficult 


5.  If  you  found  any  of  the  preceding  steps  in  the  Master  Index 
procedure  difficult  to  perform,  was  it  because  of  difficulty 
in  (check  one  or  more  below) : 

A.  Translating  problem  parameters  into  Index  search  terms. 
B.  Overall  organization  of  the  Master  Index. 

C.  Format  of  the  Master  Index  Table  of  Contents. 

D.  Finding  the  correct  search  term  in  the  left  hand  margin 
of  the  Master  Index. 

E.  Understanding  the  relationship  within  the  Master  Index 
Table  of  Contents. 

F.  Finding  the  correct  table  number  in  the  Master  Index 
Table  oi  Contents. 


COMMENTS : 


Stop  Time 


■5 


■i 


1 


| 
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Problem  2 


Start  Time 


As  a member  of  the  Reliability  Analysis  Group,  you  must 
estimate  the  availability  of  a new  fire  control  system  radar 
transmitter  now  under  development.  Since  availability  is 
determined  by  (among  other  factors)  unscheduled  organizational 
(troubleshooting)  time,  you  must  include  troubleshooting  time  in 
your  calculations.  However,  no  data  on  the  new  transmitter  are 
available.  You  therefore  attempt  to  predict  this  troubleshooting 
time  on  the  basis  of  troubleshooting  data  for  previous  transmit- 
ters, since  the  new  transmitter  is  not  substantially  advanced 
over  those  of  the  present  generation  (FplllA,  A-7D) . What  would 
you  estimate  as  average  troubleshooting  time  for  the  new 
transmitter? 

The  data  relevant  to  this  problem  can  be  found  in  Table 
1.30-9.17. 


ANSWER: 

1.  How  easy  or  difficult  was  this  problem  to  solve? 


I 1 1 1 I 

Very  Easy  Neither  Easy  Difficult  Very 

Easy  Nor  Difficult  Difficult 

2.  How  certain  are  you  about  the  correctness  of  your  answer? 


f 1 1 1 I 

Completely  Very  Neither  Very  Very  Completely 

Certain  Certain  Certain  nor  Very  Uncertain  Uncertain 

Uncertain 

3.  If  this  problem  was  difficult  to  solve  or  you  are  uncertain 
about  the  answer,  indicate  why  by  checking  one  or  more  of  the 
following  and  explain  under  COMMENTS: 

A.  Didn't  understand  problem. 

B.  Not  the  kind  of  problem  I am  familiar  with. 

C.  Did  not  have  enough  time  to  solve. 

D.  Not  enough  data  in  table  to  solve  problem. 

E.  Data  in  table  was  irrelevant  to  problem. 

F.  Data  difficult  to  extract  from  table. 

G.  Information  in  table  too  verbal. 

H.  Data  not  current. 
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3.  (continued) 

I.  Organization  of  table  confusing. 

J.  Implications  of  data  in  table  not  clearly  pointed  out. 

K.  Too  much  information  presented  in  table. 

L.  Too  much  irrelevant  information  in  table. 

4.  In  your  work  have  you  encountered  problems  similar  to  the  one 
you  have  just  solved? 


Constantly 


Very 

Often 


Sometimes 


Rarely 


Never 


5.  If  in  your  work  you  encountered  a problem  similar  to  the  one 
you  have  just  solved,  would  the  material  in  the  tables  hava 
been  useful  in  solving  that  problem? 


Extremely 


Very 

Useful 


Moderately 

Useful 


Slightly 

Useful 


1 

No  Use 

At  All 


6.  What  information  do  you  currently  use  to  solve  such  a problem? 
Check  one  or  more: 

A.  Information  is  unavailable. 

B.  Government  reports. 

C.  Company  historical  data. 

D.  Logic. 


E.  Textbooks. 


F.  Other  (describe) 


7.  For  this  problem  how  useful  would  your  own  data  sources  be 
compared  to  the  handbook? 


Much  More 
Useful 


More 

Useful 


Just  as 
Useful 


Slightly 

Useful 


Much  Less 
Useful 
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8.  For  this  problem  how  accessible  would  your  own  data  sources 
be  compared  to  the  handbook? 


Much  More  More  Equally  Less  Much  Less 

Accessible  Accessible  Accessible  Accessible  Accessible 


Problem  3 


Start  Time 


What  is  the  relationship  between  3ABR32331  training  time  on 
fire  control  indicator  scopes  and  skill  level  3?  Find  the  number 
of  the  table  describing  this  relationship  by  using  either  the 
Master  Index  or  the  Alphabetical  Index. 

ANSWER:  Table 


1.  How  easy  or  difficult  was  this  problem  to  solve? 


I I 1 1 1 

Very  Easy  Neither  Easy  Difficult  Very 

Easy  Nor  Difficult  Difficult 

2.  Which  indexing  system  did  you  use  to  find  the  table? 

A.  Master  Index 

B.  Alphabetical  Index. 

C.  Tried  Master  Index  first,  then  Alphabetical  Index. 

D.  Tried  Alphabetical  Index  first,  then  Master  Index. 


3.  Why  did  you  select  the  index  you  used? 

A.  is  faster. 

B.  is  easier. 

C • leads  more  directly  to  correct  table. 

P.  Other  (describe)  

4.  With  what  word  or  words  did  you  enter  the  index  first? 


If  you  used  additional  terms,  what  were  they? 


COMMENTS : 


Stop  Time 
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Problem  4 


Start  Time 


M 


The  question  of  the  availability  in  the  1980's  of  3,  5,  7 and 
9 skill  levels  f6r  career  fields  321XX  through  326XX  arises.  Use 
either  the  Master  Index  or  the  Alphabetical  Index  to  find  the 
table  that  will  provide  the  needed  data. 


ANSWER:  Table 


1.  How  easy  or  difficult  was  this  problem  to  solve? 


Neither  Easy 
Nor  Difficult 


Difficult 


Very 

Difficult 


2.  Which  indexing  system  did  you  use  to  find  the  table? 

A.  Master  Index. 

B.  Alphabetical  Index 

C.  Tried  Master  Index  first,  then  Alphabetical  Index. 

D.  Tried  Alphabetical  Index  first,  then  Master  Index. 


3.  Why  did  you  select  the  index  you  used? 
A.  is  faster. 


is  easier. 


leads  more  directly  to  correct  table. 


D.  Other  (describe) 


4.  With  what  word  or  words  did  you  enter  the  index  first? 


If  you  used  additional  terms,  what  were  they? 


Problem  5 Start  Time 


DO  NOT  USE  YOUR  HANDBOOK  TO  SOLVE  THIS  PART  OF  THE  PROBLEM 

You  are  part  of  a project  team  developing  policies  for 
training,  promotion  and  retention  of  enlisted  technicians.  Your 
specific  responsibility  is  avionics  personnel.  Assuming  that 
future  requirements  for  and  availability  of  avionics  maintenance 
manpower  will  be  the  same  as  those  of  the  current  period  (circa 
1970) , in  what  grade  levels  (E-4  through  E-9)  will  there  be 
shortages? 


ANSWER:  Check  one  or  more  below: 


E-4  E-5  E-6  E-7 


E-8  E-9 


1.  How  easy  or  difficult  was  this  problem  to  solve? 


I 1 1 1 1 

Very  Easy  Neither  Easy  Difficult  Very 

Easy  Nor  Difficult  Difficult 


2.  How  certain  are  you  about  the  correctness  of  your  answer? 


Completely 

Certain 


Very 

Certain 


Neither  Very 
Certain  nor  Very 
Uncertain 


Very 

Uncertain 


Completely 

Uncertain 


Problem  5 (continued) 


Now  open  your  handbook  to  Table  II . 2- 31 . 1 which  contains  data 
relevant  to  this  problem.  Analyze  the  table  to  confirm  your 
previous  answer  or  to  find  a better  one.  If  you  should  change 
your  mind  about  that  previous  answer,  do  not  revise  it. 


ANSWER:  Check  one  or  more  below: 

E-4  E-5  E-6  E-7  E-8  E-9 

1.  How  easy  or  difficult  was  this  problem  to  solve  with  the 
handbook? 


I 1 1 1 1 

Very  Easy  Neither  Easy  Difficult  Very 

Easy  Nor  Difficult  Difficult 


2>.  How  certain  are  you  now  about  the  correctness  of  your  answer? 

Completely  Very  Neither  Very  Very  Completely 

Certain  Certain  Certain  nor  Very  Uncertain  Uncertain 

Uncertain 

3.  If  this  problem  was  difficult  to  solve  or  you  are  uncertain 
about  the  answer,  indicate  why  by  checking  one  or  more  of  the 
following  and  explain  under  COMMENTS: 

A.  Didn't  understand  problem. 

B.  Not  the  kind  of  problem  I am  familiar  with. 

C.  Did  not  have  enough  time  to  solve. 

D.  Not  enough  data  in  table  to  solve  problem. 

E . Problem  not  realistic. 

F.  Data  in  table  was  irrelevant  to  problem. 

G.  Data  difficult  to  extract  from  table. 

H.  Can't  solve  problem  without  handbook. 

I.  Information  in  table  too  verbal. 

J.  Data  not  current. 

K.  Organization  of  table  confusing. 


69 


3. 


(continued) 


L.  Implications  of  data  in  table  not  clearly  pointed  out.  ^ 

M.  Too  much  information  presented  in  table. 

N.  Too  much  irrelevant  information  in  table. 

0.  Other  (describe) 


4. 


In  your  work  have  you  encountered  problems  similar 
you  have  just  solved? 


to  the  one 


Constantly 


Very 

Often 


Sometimes 


Rarely 


Never 


5. 


“„i"^ur  w°rk  you  encountered  a problem  similar  to  the  one 
you  have  just  solved,  would  the  material  in  the  tables  have 
been  useful  in  solving  that  problem? 


Extremely  Very  Moderately 

Useful  Useful  Useful 


Slightly  No  Use 

Useful  At  All 


6'  3HrtiS£rSJti£.!° you  cutr<intly use  10  *°1to  =«<=»  * probi^ 


A*  Information  is  unavailable. 


B. 

Government  reports. 

C. 

Company  historical  data. 

D. 

Logic. 

E. 

Textbooks . 

F. 

Other  (describe) 

7. 


For  this  problem  how  useful  would 
compared  to  the  handbook? 


your  own  data 


sources  be 


Much*  More  More  Just  as 

Useful  Useful  Useful 


/I  1 1 

uess  Much  Less 

Useful  Useful 
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For  this  problem  how  accessible  would  your  own  data  sources 
be  compared  to  the  handbook?  — 


Much  More  More 

Accessible  Accessible 


Equally  Less  Much  Less 

Accessible  Accessible  Accessible 


COMMENTS : 


Stop  Time 


Problem  6 


Start  Time 


T 


f 


H 
1 1 


The  X company  is  bidding  on  the  development  of  a new 

generation  fire  control  radar  subsystem.  To  determine  total 
development  cost  you  must  predict  training  time  for  level  3 
mechanics  on  the  new  system.  There  may  be  a relationship  between 
the  cost  of  developing  the  system  and  training  time,  so  you  need 
data  relating  training  time  to  acquisition  cost.  However,  all 
you  have  is  a best  estimate  of  the  subsystem  cost,  $225,000.  If 
the  relationship  between  cost  and  training  time  is  correct,  how 
long  should  training  time  (in  weeks)  be? 

The  data  relevant  to  this  problem  can  be  found  in  Table 
1.19-3.1. 

ANSWER:  

1.  How  easy  or  difficult  was  this  problem  to  solve? 


1 

! 

I 


I I I T 1 

Very  Easy  Neither  Easy  Difficult  Very 

Easy  Nor  Difficult  Difficult 

2.  How  certain  are  you  about  the  correctness  of  your  answer? 

Completely  Very  Neither  Very  Very  Completely 

Certain  Certain  Certain  nor  Very  Uncertain  Uncertain 

Uncertain 

3.  If  this  problem  was  difficult  to  solve  or  you  are  uncertain 
about  the  answer,  indicate  why  by  checking  any  or  all  of  the 
following  and  explain  under  COMMENTS: 

tA.  Didn't  understand  problem. 

B.  Not  the  kind  of  problem  I am  familiar  with. 

C.  Did  not  have  enough  time  to  solve. 

D.  Not  enough  data  in  table  to  solve  problem. 

E.  Problem  not  realistic. 

F.  Data  in  table  was  irrelevant  to  pro!  lem. 

G.  Data  difficult  to  extract  from  table. 

H.  Information  in  table  too  verbal. 

I.  Data  not  current. 
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3.  (continued) 

J.  Organization  of  table  confusing. 

K.  Implications  of  data  in  table  not  clearly  pointed  out. 

L.  Too  much  information  presented  in  table. 

M.  Too  much  irrelevant  information  in  table. 

N.  Other  (describe) 

4.  In  your  work  have  you  encountered  problems  similar  to  the  one 
you  have  just  solved? 


Constantly 


Very 

Often 


Sometimes 


Rarely 


Never 


5.  If  in  your  work  you  encountered  a problem  similar  to  the  one 
you  have  just  solved,  would  the  material  in  the  tables  have 
been  useful  in  solving  that  problem? 


Extremely 

Useful 


Very 

Useful 


Moderately 

Useful 


Slightly 

Useful 


No  Use 
At  All 


6.  What  information  do  you  currently  use  to  solve  such  a problem? 
Check  one  or  more: 

A.  Information  is  unavailable. 

B.  Government  reports. 

C.  Company  historical  data. 

D.  Logic. 


E.  Textbooks. 


F.  Other  (describe) 


7.  For  this  problem  how  useful  would  your  own  data  sources  be 
compared  to  the  handbook? 


Much  More 
Useful 


More 

Useful 


Just  as 
Useful 


Less 
Use'f  ul 


Much  Less 
Useful 
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8.  For  this  problem  how  accessible  would  your  own  data  sources 
be  compared  to  the  handbook? 


Much  More 
Accessible 


More 

Accessible 


Equally 

Accessible 


Less 

Accessible 


Much  Less 
Accessible 


COMMENTS : 


Stop  Time 
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Problem  7 


Start  Time 


In  the  process  of  developing  a new  generation  fire  control 
system  you  are  required  to  predict  the  percent  of  time  that  will 
be  spent  by  3,  5 and  7, skill  level  maintenance  men  performing 
general  electronic  maintenance  and  repair  jobs  on  the  new  system. 
Find  the  number  of  the  table  providing  these  data  by  using  the 
Master  Index. 


ANSWER:  Table  

1.  How  easy  or  difficult  was  the  problem  to  solve? 


Neither  Easy 
Nor  Difficult 


Difficult 


Very 

Difficult 


2.  How  easy  was  it  to  use  the  Master  Index  in  finding  the  correct 
table? 


Neither  Easy 
Nor  Difficult 


Difficult 


Very 

Difficult 


3.  With  what  word  or  words  did  you  enter  the  Index  first? 


If  you  used  additional  terms,  what  were  they? 


4.  How  easy  was  it  to  perform  the  individual  steps  of  the  Master 
Index  procedure? 

A.  To  determine  section  number  from  Master  Index: 


Neither  Easy 
Nor  Difficult 


Difficult 


Very 

Difficult 


B.  To  find  the  appropriate  index  number  from  the  Master  Index 
Table  of  Contents? 


Neither  Easy 
Nor  Difficult 


Difficult 


Very 

Difficult 


4 


(continued) 


! 


! 


1 


c. 


To  determine  the  table  number  from  the  Master  Index  Table 
of  Contents? 


i i i i i 

Very  Easy  Neither  Easy  Difficult  Very 

Easy  Nor  Difficult  Difficult 

5.  If  you  found  any  of  the  preceding  steps  in  the  Master  Index 
procedure  difficult  to  perform,  was 'it  because  of  difficulty 
in  (check  one  or  more  below) : 

A.  Translating  problem  parameters  into  Index  search  items. 

B.  Overall  organization  of  the  Master  Index. 

C.  Format  of  the  Master  Index  Table  of  Contents. 

D.  Finding  the  correct  search  term  in  the  left  hand  margin 

of  the  Master  Index. 

E.  Understanding  the  relationship  within  the  Master  Index 

Table  of  Contents. 

j 

F.  Finding  the  correct  table  number  in  the  Master  Index  Table 

of  Contents. 


Problem  8 


Start  Time 


What  is  the  relationship  between  radar  subsystems  logistics 
support  costs  and  unscheduled  maintenance  manhours?  Find  the 
number  of  the  table  describing  this  relationship  by  using  the 
Alphabetical  Index. 


ANSWER:  Table 


1.  How  easy  or  difficult  was  the  problem  to  solve? 


I 1 ^ 1 1 

Very  Easy  Neither  Easy  Difficult  Very 

Easy  Nor  Difficult  Difficult 


2.  How  easy  was  it  to  use  the  Alphabetical  Index  to  find  the 
correct  table? 


(— 1 1 1 1 

Very  Easy  Neither  Easy  Difficult  Very 

Easy  Nor  Difficult  Difficult 


3.  With  what  word  or  words  did  you  enter  the  Index  first? 
If  you  used  additional  terms,  what  were  they?  


4.  If  you  found  the  Alphabetical  Index  difficult  to  use,  was  it 
because  (check  one  or  more) : 

A.  The  listings  are  incomplete. 

B.  The  listings  are  not  logically  arranged. 

C.  The  listings  are  not  sufficiently  detailed. 

D.  Too  much  cross-referencing  required  to  find  correct  listing. 

E.  Other  (describe  fully) . 


COMMENTS : 


Stop  Time 


Problem  9 


Start  Time 


I 


u 


DO  NOT  USE  YOUR  HANDBOOK  TO  SOLVE  THIS  PROBLEM 

In  developing  future  manpower  cost  projections  for  the  Air 
Force,  it  is  necessary  to  predict  the  cost  of  training  a man  in 
system  maintenance  of  fire  control  systems  (3ABR32331).  The  cost 
per  student  for  this  type  of  training  from  1959  to  1968  (latest 
data  available)  averaged  $8,191.  Assuming  no  significant  changes 
from  1968  costs,  what  do  you  estimate  the  cost  will  be  in  1982? 

ANSWER: 


1.  How  easy  or  difficult  was  this  problem  to  solve? 


Very 

Easy 


Easy 


Neither  Easy 
Nor  Difficult 


1 

Difficult 


Very 

Difficult 


2.  How  certain  are  you  about  the  correctness  of  your  answer? 


Completely  Very  Neither  Very  Very  Completely 

Certain  Certain  Certain  nor  Very  Uncertain  Uncertain 

Uncertain 
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Problem  9 (continued) 

Now  open  your  handbook  to  Table  1.22-2.1  which  contains  data 
relevant  to  this  problem.  Analyze  the  table  to  confirm  your 
previous  answer  dr  to  secure  a new  one.  If  you  should  change 
your  mind  about  that  previous  answer,  do  not  revise  it. 

ANSWER: 


1.  How  easy  or  difficult  was  this  problem  to  solve  with  the 
handbook? 


i i ; — i 1 1 

Very  Easy  Neither  Easy  Difficult  Very 

Easy  Nor  Difficult  Difficult 

2.  How  certain  are  you  now  about  the  correctness  of  your  answer? 

I 1 f 1 1 

Completely  Veiy  Neither  Very  Very  Completely 

Certain  Certain  Certain  nor  Very  Uncertain  Uncertain 

Uncertain 

3.  If  this  problem  was  difficult  to  solve  cr  you  are  uncertain 
about  the  answer,  indicate  why  by  checking  one  or  more  of  the 
following  and  explain  under  COMMENTS: 

A.  Didn't  understand  problem. 

B.  Not  the  kind  of  problem  I am  familiar  with. 

C.  Did  not  have  enough  time  to  solve. 

D.  Not  enough  data  in  table  to  solve  problem. 

E.  Problem  not  realistic. 

F.  Data  in  table  was  irrelevant  to  problem. 

G.  Data  difficult  to  extract  from  table. 

H.  Can't  solve  problem  without  handbook. 

I . Information  in  table  too  verbal. 

J.  Data  not  current. 

If . Organization  of  table  confusing. 

L.  Implications  of  date,  in  table  not  clearly  pointed  out. 

*.  Too  much  information  presented  in  table. 
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(continued) 


N.  Too  much  irrelevant  information  in  table. 

0.  Other  (describe)  

4.  In  your  work  have  you  encountered  problems  similar  to  the  one 
you  have  just  solved? 


Constantly 


Very  Sometimes  Rarely 

Often 


Never 


5.  If  in  your  work  you  encountered  a problem  similar  to  the  one 
you  have  just  solved,  would  the  material  in  the  tables  have 
been  useful  in  solving  that  problem. 


I I ' I 1 I 

Extremely  Very  Moderately  Slightly  No  Use 

Useful  Useful  Useful  Useful  At  All 

6.  ##hat  information  do  you  currently  use  to  solve  such  problems? 
Check  one  or  more: 

A.  Information  is  unavailable. 

B.  Government  reports. 

C.  Company  historical  data. 

D.  Logic. 

_E.  Textbooks. 

F . . Other  (describe ) 

7.  For  this  problem  how  useful  would  your  own  data  sources  be 
compared  to  the  handbook? 


Much* More  More  Just  as  Less  Much  1 Less 

Useful  Useful  Useful  Useful  Useful 

8.  For  this  problem  how  accessible  would  your  own  data  sources 
be  compared  to  the  handbook? 


Much  More  More 

Accessible  Accessible 


Equally 

Accessible 


1 

Less 

Accessible 


r 

Much  Less 
Accessible 
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Problem  9 (continued) 


Problem  10 


Start  Time 


The  Air  Force  has  specified  that  maximum  unit  replacement 
time  for  a new  avionics  system  under  development  cannot  exceed  6 
hours . Your  task  is  to  predict  actual  replacement  time.  It  is 
a reasonable  hypothesis  that  maintenance  time  is  at  least 
partially  determined  by  hardware  design  characteristics,  in  this 
case  the  number  of  components  that  must  be  manipulated.  The  new 
syste...  is  composed  of  30  major  parts  that  must  be  handled  in 
order  to  repair  it.  Estimate  the  unit  replacement  time  for  this 
system  and,  if  it  exceeds  Air  Force  requirements,  the  number  of 
components  that  will  have  to  be  eliminated. 

The  data  relevant  to  this  problem  can  be  found  in  Table 
III. 5-36. 5. 

ANSWER:  

1.  How  easy  or  difficult  was  this  problem  to  solve? 


I 1 1 1 1 

Very  Easy  Neither  Easy  Difficult  Very 

Easy  Nor  Difficult  Difficult 

2.  How  certain  are  you  about  the  correctness  of  your  answer? 


I 1 — I 1 1 

Completely  Very  Neither  Very  Very  Completely 

Certain  Certain  Certain  nor  Very  Uncertain  Uncertain 

Uncertain 

3.  If  this  problem  was  difficult  to  solve  or  you  are  uncertain 
about  the  answer,  indicate  why  by  checking  one  or  more  of 
the  following  and  explain  under  COMMENTS: 

A.  Didn't  understand  problem. 

b.  Not  the  kind  of  problem  I am  familiar  with. 

C.  Did  not  have  enough  time  to  solve. 

D.  Not  enough  data  in  table  to  solve  problem. 

E.  Problem  not  realistic. 

F.  Data  in  table  was  irrelevant  to  problem. 

G.  Data  difficult  to  extract  from  table. 

H.  Information  in  table  too  verbal. 

I.  Data  not  current. 
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3. 


(continued) 


_J.  Organization  of  table  confusing. 

_K.  Implications  of  data  in  table  not  clearly  pointed  out. 
_L.  Too  much  information  presented  in  table. 

_M.  Too  much  irrelevant  information  in  table. 

N.  Other  (describe) 


4.  In  your  work  have  you  encountered  problems  similar  to  the  one 
you  have  just  solved? 


i | 1 1 1 

Constantly  Very  Sometimes  Rarely  Never 

Often 

5.  If  in  your  work  you  encountered  a problem  similar  to  the  one 
you  have  just  solved,  would  the  material  in  the  tables  have 
been  useful  in  solving  that  problem? 


I 1 ! r 1 

Extremely  Very  Moderately  Slightly  No  Use 

Useful  Useful  Useful  Useful  At  All 

6.  What  information  do  you  currently  use  to  solve  such  problems? 
Check  one  or  more: 

A.  Information  is  unavailable. 

B.  Government  reports. 

C.  Company  historical  data. 

D.  Logic. 

E.  Textbooks. 

F.  Other  (describe) 

7.  For  this  problem  how  useful  would  your  own  data  sources  be 
compared  to  the  handbook? 


I 

Much  More 
Useful 


More  Just  as 

Useful  Useful 


1 1 

Less  Much  Less 

Useful  Useful 
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8.  For  this  problem  how  accessible  would  your  own  data  sources 
be  compared  to  the  handbook? 


I r i i i 

Much  More  More  Equally  Less  Much  Less 

Accessible  Accessible  Accessible  Accessible  Accessible 


j 


COMMENTS 


Problem  11 


Start  Time 


During  the  development  of  a new  generation  avionics  system 
you  must  decide  between  two  designs  that  differ  in  their  hardware 
characteristics . Since  one  system  requires  more  steps  in 
functional  checkout  than  the  other,  this  may  create  more  errors 
in  the  technician's  performance.  Find  the  table  that  relates 
number  of  steps  in  functional  checkout  to  number  of  errors.  Use 
the  Alphabetical  Index  for  this  purpose. 

ANSWER:  Table  

1.  How  easy  or  difficult  was  the  problem  to  solve? 


Very 

Easy 


Easy  Neither  Easy 

Nor  Difficult 


1 

Difficult 


Very 

Difficult 


2.  How  easy  was  it  to  use  the  Alphabetical  Index  to  find  the 
correct  table? 


I 1 1 r—  1 

Very  Easy  Neither  Easy  Difficult  Very 

Easy  Nor  Difficult  Difficult 

3.  With  what  word  or  words  did  you  enter  the  Index  first?  

i 

' 

/ If  you  used  additional  terms,  what  were  they?  


4.  If  you  found  the  Alphabetical  Index  difficult  to  use,  was  it 
because  (check  one  or  more) : 

A.  The  listings  are  incomplete. 

B.  The  listings  are  not  logically  arranged. 

C . The  listings  are  not  sufficiently  detailed. 

D.  Too  much  cross-referencing  required  to  find  correct  listing. 


t-i 


Problem  12 


Start  Time 


In  planning  the  integrated  logistics  system  for  a new  fire 
control  radar  subsystem  you  must  estimate  the  number  of  mainte- 
nance manhours  (MMH)  per  1000  flight  hours  (FH) . Recent  data 
suggest  that  there  is  a relationship  between  number  of  MMH  and 
number  of  line  replaceable  units  (LRUs) . The  new  subsystem  will 
contain  12  LRUs.  What  is  your  estimate  of  the  expected  MMH/1000FH 
(unscheduled  organizational  maintenance)? 

The  data  relevant  to  this  problem  can  be  found  in  Table 
1.11-4.1. 


ANSWER:  

1.  How  easy  or  difficult  was  this  problem  to  solve? 


I ! 


8.  For  this  problem  how  accessible  would  your  own  data  sources 
be  compared  to  the  handbook? 


Much  More 
Accessible 


COMMENTS : 


More 

Accessible 


Equally 

Accessible 


Less 

Accessible 


Much  Less 
Accessible 


Stop  Time 


r 


1 


I 


i 

1 


i 

i 

f ' 

I I 

l 


: ‘ 

I l 


| 


2.  Did  the  tables  provide  as  much  data  as  you  would  want  in  a 
handbook  of  this  type? 


I I 1 1 1 

Much  More  More  than  Just  Enough  Not  as  Much  Seriously 

than  Desirable  Enough  as  it  Should  Lacking 

3.  If  the  tables  did  not  contain  enough  data,  how  much  more  data 
and  of  what  type  would  you  want?  


4.  How  useful  would  an  improved  version  of  this  type  of  handbook 
be  in  the  work  you  do? 

I ! 1 ( 1 

Extremely  Very  Moderately  Slightly  Not  Useful 

Useful  Useful  Useful  Useful  At  All 

5.  For  which  of  the  following  types  of  problems  could  such  a 
handbook  be  used  effectively?  Check  one  or  more  and  rank 
those  you  check. 

Avionics  and  fire  control  system  historical  data. 

Acquisition  cost  of  avionics  and  fire  control  systems. 

Equipment  desigh  tradeoffs. 

Description  of  avionics/fire  control  tasks/ jobs. 

Human  error  prediction. 

Manpower  costs. 

Personnel  selection  criteria. 

Manpower  skill  and/or  availability. 

Training  costs  and  training  time. 

Logistics  costs  and  tradeoffs. 

Maintainability  and  maintenance  performance  times. 

Manpower  life  cycle  costs. 

Prediction  of  technician  performance. 

Fire  control  design  characteristics. 
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. Assume  that  you  had  to  solve  the  kinds  of  problems  described 
in  the  immediately  previous  question,  how  influential  would 
an  improved  handbook  of  this  type  be  in  affecting  your  system 
design  decisions? 


I I I l 1 

Extremely  Very  Moderately  Slightly  No  Influence 

Influential  Influential  Influential  Influential  At  All 

7.  What  would  you  recommend  to  improve  the  handbook?  Check  one 
or  more  of  the  following  and  rank  those  you  check. 

Improve  overall  handbook  organization. 

Improve  organization  of  individual  tables. 

‘Make  data  more  current. 

Increase  amount  of  data  in  tables. 

Provide  more  data  points. 

Reduce  verbiage  in  tables. 

Present  data  in  more  quantitative  form. 

Make  index  easier  to  use. 

Cover  more  subject  areas. 

Reduce  amount  of  material  in  tables. 

Indicate  uses  of  data  more  effectively. 

Broaden  data  sources. 

Provide  more  graphic  illustrations. 

Make  instructions  for  indexing  system  clearer. 

8.  How  often  have  you  used  human  resources  data  in  your  work? 
Remember  that  human  resources  data  are  ,data  relating  to 
personnel  availability,  training,  use,  cost,  etc. 

I 1 1 1 “ 1 

Constantly  Often  Sometimes  Rarely  Never 
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9.  If  you  have  used  such  data 


A.  What  type?  

B.  Where  did  you  get  it? 

C.  For  what  types  of  problems?  (Use  list  in  question  5 as 

guide)  


How  useful  did  you  find  these  data? 


Extremely 

Useful 


Very 

Useful 


Moderately 

Useful 


Slightly 

Useful 


Not  Useful 
At  All 


10.  How  important  do  you  consider  human  resources  and  human  factors 
in  operation  and  maintenance  of  avionics  and  fire  control 
equipment? 


More  Important  More  Important  As  Important  Less  Important  Completely 

than  Anything  than  Physical  as  Physical  than  Physical  Unimportant 

Else  (Engineering)  Factors  Factors 

Factors 

11.  How  important  do  you  consider  human  resources  data  in  affecting 
system  design? 


Overwhelmingly  More  Important  As  Important  Less  Important  Completely 
Important  than  Other  as  Any  Other  than  Other  Unimportant 

Data  Data  Data 
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INTRODUCTION  TO  APPENDICES  B AND  C 
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i 


i 

> i 

► ; 

f ; 


The  data  items  listed  in  these  appendices  are  defined 
as  follows: 

Specialty:  Participant's  background. 

Years  of  experience:  Years  worked  in  specialty. 

% problems  correctly  solved:  Mean  percentage  of  the  12 

problems  correctly  solved. 

Solution  time:  Time  taken  by  participant  to  solve  problem. 

The  following  items  are  rating  measures: 

Problem  difficulty:  Rating  of  problem  difficulty. 

Solution  certainty:  Rating  of  participant's  confidence  in 

the  correctness  of  his  problem 
solution. 


Problem  fidelity: 

HR  data  utility: 
Own  data  utility: 


Rating  of  the  similarity  of  questionnaire 
problems  to  those  the  participant  deals 
with  ordinarily. 

Rating  of  prototype  handbook  usefulness 
in  solving  the  problem. 

Rating  of  the  usefulness  of  participant's 
own  data  sources  compared  with  the 
prototype  handbook. 


Own  data  accessibility:  Rating  of  the  accessibility  of  the 

participant's  own  data  sources 
compared  with  that  of  the  prototype 
handbook . 

Hdbk  utility:  Rating  of  the  utility  of  an  improved  version 

of  the  prototype  handbook. 


Hdbk  influence: 


Use  of  HR  data: 


HR  importance: 


Rating  of  the  influence  an  improved 
prototype  handbook  would  have  on  system 
design. 

Rating  of  how  often  the  participant  has 
used  HR  data  in  his  work,. 

Rating  of  the  importance  of  HR  and  HR  data 
in  system  design. 


Data  solution  times  and  questions  1 through  8 represent 
the  mean  and  standard  deviation  (SD)  of  each  participant's 
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responses  over  the  12  problems.  Since  items  A-D  were 
presented  only  once  (at  the  conclusion  of  the  questionnaire) 
only  the  single  response  is  presented.  Although  data  for 
solution  times  are  expressed  in  tenths  of  minutes,  this 
results  simply  from  averaging  the  time  data;  actual  times 
were  reported  by  participants  only  to  the  nearest  minute. 
Blanks  in  Appendix  B result  from  individual  failures  to 
respond  to  a particular  question.  The  original  computations 
were  carried  out  to  two  decimal  places,  but  for  simplicity 
the  data  in  these  appendices  have  been  rounded  off  to  one 
digit. 


APPENDIX  B 


PARTICIPANT  RESPONSES  TO  THE  QUESTIONNAIRE 


Data  Category 


pecialty 


ears  of  Experience 


% Problems  Correctly  Solved 


olution  Time  (Min.) 


5roblem  Difficulty  (q.  1) 


olution  Certainty  (q.  2) 


’roblem  Fidelity  (q.  4) 


R Data  Utility  (q.  5) 


wn  Data  Utility  (q.  7) 


Participants 


12  3 

Maint.  Design  Avion. 


*6. 0/4. 9 4. 6/3. 7 5. 7/3. 4 


2. 7/1. 5 3.2/1. 8 3. 6/1. 4 


3. 4/1. 8 3. 7/2.2  4. 0/0. 8 


6. 0/1. 5 3. 7/1.1  4. 0/0. 9 


1.7/0. 8 3. 0/1. 2 3. 5/1. 2 


5. 0/1. 7 4. 0/0.0  3. 5/0. 8 


4 

Avion. 


8 


.92 


6. 9/4. 3 


3. 7/2. 2 


1 . 7/1 . 5 


7. 0/0.0 


wn  Data  Accessibility  (q.  8)|  5. 5/0.0  5. 2/0. 7 4. 2/1. 8 


(A)  Hdbk  Utility 


(B)  Hdbk  Influence 


(C)  Use  of  HR  Data 


(D)  HR  Importance 


— 

Specialty 

5 

Avion. 

fears  of  Experience 

20 

i Problems  Correctly  Solved 

.83 

Solution  Time  (Min.) 

4. 7/2. 6 

’roblem  Difficulty  (q.  1) 

3. 2/1.0 

Solution  Certainty  (q.  2) 

4. 3/0. 7 

Problem  Fidelity  (q.  4) 

5. 3/1.6 

♦First  number  = Mean 

Second  number  = Standard  Deviation  of  Mean 
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Appendix  B (continued) 


Participants 


r (continued) 


R Data  Utility  (q.  5) 


wn  Data  Utility  (q.  7) 


n Data  Accessibility  (q.  8) 


(A)  Hdbk  Utility 


(B)  Hdbk  Influence 

(C)  Use  of  HR  Data 

(D)  HR  Importance 

pecialty 


8 

Maint. 


3. 5/0.6  3. 4/1. 3 3. 8/0. 3 4. 0/2.1 


2. 5/2.1 


6. 3/0. 7 5. 5/1.1 


6. 5/0. 6 5. 5/1.1 


Solution  Time  (Min.) 


‘roblem  Difficulty  (q.  1) 


Solution  Certainty  (q.  2) 


Problem  Fidelity  (q.  4) 


HR  Data  Utility  (q.  5) 


wn  Data  Utility  (q.  7) 


n Data  Accessibility  (q.  8) 


(A)  Hdbk  Utility 


(B)  Hdbk  Influence 


(C)  Use  of  HR  Data 


(D)  HR  Importance 


3. 5/1.0 


3. 5/1.0 


3. 4/0. 8 


5. 2/0. 7 


3. 7/1. 3 


4. 3/2. 2 


5. 5/1.1 


5.5 


5.5 


6. 2/3.1 


4. 0/1. 6 


4. 2/1.0 


4. 7/0. 8 


3. 2/1. 3 


4. 2/2.0 


5. 8/0. 6 


2.5 


2.5 


2.5 
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Data  Category 

Participants 

13 

14 

15 

16 

Specialty 

Design 

Crew 

HF 

Maint . 

fears  of  Experience 

10 

21 

18 

18 

% Problems  Correctly  Solved 

.67 

.46 

T 75 

.71 

Solution  Time  (Min.) 

6. 9/3. 5 

6. 3/5. 3 

6. 7/3. 3 

4. 6/2. 5 

Problem  Difficulty  (q.  1) 

4. 5/1. 6 

3. 5/1. 2 

4. 2/1. 2 

4. 1/0. 6 

Solution  Certainty  (q.  2) 

5. 5/1. 8 

4. 0/1.1 
— ■ " — " 

4. 8/1. 4 

3. 5/0. 5 

’roblem  Fidelity  (q.  4) 


R Data  Utility  (q.  5) 


wn  Data  Utility  (q.  7) 


wn  Data  Accessibility  (q.  8) 


(A)  Hdbk  Utility 


(B)  Hdbk  Influence 


(C)  Use  of  HR  Data 


D)  HR  Importance 


6. 5/0.8 


4. 7/1. 8 


5. 5/0.0 


5.5 


4.0 


4.0 


7. 0/0.0  5.2/0. 8 


3. 0/0. 8 3. 5/1.6  4. 5/1. 3 


4. 0/0. 9 4. 5/1. 9 3. 9/2. 2 


5. 5/0.0  5. 6/0. 5 3. 3/0. 7 


5.5  2.5 


4.0  4.0 


4.0  2.6  3.3 


5.5  2.5 


pecialty 


fears  of  Experience 


'%  Problems  Correctly  Solved 


17  18  19 

Train.  Ops. An.  Design 


Solution  Time  (Min.) 

4. 4/1. 8 

6. 2/2. 6 

7. 2/2. 6 

8. 5/4.0 

Problem  Difficulty  (q.  1) 

3. 9/0. 9 

2. 7/2.0 

3. 4/1. 4 

3. 6/1. 7 

Solution  Certainty  (q.  2) 

4. 1/1.0 

3.6/1. 7 

2. 9/1. 2 

3. 2/1.1 

Problem  Fidelity  (q.  4) 

4. 8/1. 3 

4. 0/0.0 

6. 8/0. 3 

4.2/0. 6 

HR  Data  Utility  (q.  5) 

4. 5/1.1 

3. 0/1. 5 

. 

1.9/0. 6 

4. 2/0. 6 

Own  Data  Utility  (q.  7) 

3. 4/0. 8 

5. 0/2. 3 

4. 5/1. 9 

2. 7/0. 6 

Data  Categor 


pecialty  (continued) 


wn  Data  Accessibility  (q.  8) 


(A)  Hdbk  Utility 


(B)  Hdbk  Influence 


(C)  Use  of  HR  Data 


(D)  HR  Importance 


Participants 


17 

Train. 


4. 3/0. 4 5. 0/2.0  6. 0/0. 5 2. 5/0.0 


6.2  4.0 


5.5  2.5  2.5  5.5 


3.5  4.0  6.6  4.0 


.3  4.0 


pecialty 


'ears  of  Experience 


% Problems  Correctly  Solved 


olution  Time  (Min.) 


’roblem  Difficulty  (q.  1) 


olution  Certainty  (q.  2) 


’roblem  Fidelity  (q.  4) 


R Data  Utility  (q.  5) 


wn  Data  Utility  (q.  7) 


wn  Data  Accessibility  (q.  8) 


(A)  Hdbk  Utility 


(B)  Hdbk  Influence 


(C)  Use  of  HR  Data 


(D)  HR  Importance 


9 


.58 


2.6/1. 4 


3. 9/1. 6 


4. 7/1.1 


6. 5/0. 8 


4. 2/0. 6 


5. 0/0. 8 


4. 7/0.8 


7.0 


4.0 


4.0 


26 


.46 


3. 1/2.1 


3. 4/1. 6 


4. 0/1. 6 


4. 7/1.4 


5.0/2. 3 


4. 2/1. 5 


4. 7/1. 8 


15 


.75 


2. 7/1. 3 


2. 4/1.0 


3. 8/2.0 


4. 3/1.1 


3. 0/1. 2 


4 .8/1.5 


5. 4/1.0 


2.5 


4.8 


4.0 


24 

Maint . 


10 


5 


5. 4/2. 5 


2. 9/0. 9 


3. 8/1. 5 


2. 9/0. 8 


4. 6/1.1 


4. 3/0. 7 


5.2/0. 7 


— — — 

Specialty 

25 

Maint . 

26 

Design 

27 

Design 

28 

Misc. 

Years  of  Experience 

1 

25 

12 

5 

% Problems  Correctly  Solved 

.64 

.91 

.62 

.71 
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Appendix  B (continued) 


Data  Cateqor 


pecialty  (continued) 


olution  Time  (Min.) 


’roblem  Difficulty  (q.  1) 


olution  Certainty  (q.  2) 


»roblem  Fidelity  (q.  4) 


R Data  Utility  (q.  5) 


wn  Data  Utility  (q.  7) 


9wn  Data' Accessibility  (q.  8) 


(A)  Hdbk  Utility 


(B)  Hdbk  Influence 


(C)  Use  of  HR  Data 


(D)  HR  Importance 


Participants 


»T3 


28 

Misc. 


4. 7/2. 8 5. 4/4. 2 6. 7/4.1  8. 2/4. 6 


3. 7/0.7  1.9/0. 1 3.0/1. 8 3. 5/1. 3 


3. 8/0. 9 3. 8/2.0  3. 4/2. 4 4. 9/1. 6 


4. 4/0. 9 7. 0/0.0  6. 5/1. 2 5. 9/0. 6 


3. 2/0. 5 4. 5/1. 2 2. 5/0.0  5. 7/2.0 


4. 4/1. 2 3. 5/1. 8 


7.0/0. 0 


5. 0/0. 7 3. 7/1.1  5. 5/0.0  6. 5/0. 7 


3.5  7.0  5.5  6.5 


.5  7.0  4.0  3.3 


7.0  5.5  5.5 


1 

Specialty 

29 

Misc. 

■ - ’ - 1 

30 

Misc. 

31 

Train. 

32 

AGE 

Sfears  of  Experience 

1 

11 

32 

18 

% Problems  Correctly  Solved 

.83 

.79 

.79 

.92  I 

1 

Solution  Time  (Min.) 

6. 1/3. 2 

9. 1/5.0 

5. 3/2.9 

6 .2/3.8 

Problem  Difficulty  (q.  1) 

2. 7/1. 7 

3. 5/2.0 

1. 5/1.0 

3.6/l.o] 

Solution  Certainty  (q.  2) 

3. 2/1. 4 

3. 1/1. 7 

2. 3/1. 2 

4. 2/2.0 

Problem  Fidelity  (q.  4) 

7. 0/0.0 

4. 7/1.1 

5. 7/2.0 

7. 0/0.0 


HR  Data  Utility  (q.  5) 

5. 0/1. 8 

2. 3/0.4 

2. 5/2.1 

2. 5/0.0 

wn  Data  Utility  (q.  7) 


wn  Data  Accessibility  (q.  8) 


(A)  Hdbk  Utility 


(B)  Hdbk  Influence 


6. 2/0. 8 5.7/0. 4 6. 0/0. 9 6. 


7. 0/0.0  5. 9/0. 6 5. 0/1. 7 


2.5  3.2  2.5  5.5 


Appendix  B (continued) 


Data  Categor 


Participants 


Specialty  (continued)  , 

29 

Misc. 

30 

Misc. 

31 

Train. 

32 

AGE 

(C)  Use  of  HR  Data 

4 

.0 

2.5 

2.5 

7.0 

(D)  HR  Importance 

4 

.0 

4.0 

3.2 

4.0 

Specialty 

33 

AGE 

34 

Design 

35 

Design 

36 

Avion . 

ifears  of  Experience 

16 

25 

8 

14 

% Problems  Correctly  Solved 

.82 

.60 

.87 

.75 

Solution  Time  (Min.) 

6. 2/2. 7 

5. 5/2. 7 

4. 7/1. 7 

3. 7/2. 2 

Problem  Difficulty  (q.  1) 

4. 8/1. 4 

2. 3/0. 8 

3. 0/1. 5 

3.0/1. 0 

Solution  Certainty  (q.  2) 

3. 9/1. 4 

2. 8/1. 3 

3. 4/2.0 

3. 2/1. 6 

Problem  Fidelity  (q.  4) 

5. 5/1.0 

6. 2/1. 5 

6. 5/0. 8 

5. 7/1.1 

HR  Data  Utility  (q.  5) 

4. 5/0. 9 

2. 5/0.0 

2. 5/0.0 

4. 0/1.6 

Own  Data  Utility  (q.  7) 

5. 1/1. 5 

5. 5/2.1 

i 

5. 0/1. 2 

5. 5/1. 3 

Own  Data  Accessibility  (q.  8) 

5. 8/1.1 

6. 5/0. 9 

5. 5/0.0 

5. 2/1.1 

(A)  Hdbk  Utility 

4 

.0 

5.5 

| 

4.0 

(B)  Hdbk  Influence 

■ 

.9 

2.5 

2.5 

4.0 

(C)  Use  of  HR  Data 

2 

.5 

7.0 

Mam 

4.0 

(D)  HR  Importance 

■ 

.3 

4.0 

4.0 

5.5 
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APPENDIX  C 

PARTICIPANT  RESPONSES  BY  PROBLEM 


Data  Category 

Problems 

1 

2 

— 

3 

4 

% Problems  Correctly  Solved 

*.61/. 32 

.65/. 35 

.79/. 30 

. 6 8/ . 3 2 

Solution  Time  (Min.) 

5. 3/2. 5 

5. 6/2. 9 

6. 4/4. 9 

8. 2/4.1 

Problem  Difficulty  (q.  1) 

2. 9/1. 3 

2. 9/1. 5 

3. 7/1. 6 

. 

4. 8/1.2 

Solution  Certainty  (q.  2) 

3. 4/1. 3 

j 

Problem  Fidelity  (q.  4) 

5. 2/1. 6 

1 

HR  Data  Utility  (q.  5) 

3. 3/1. 3 

Own  Data  Utility  (q.  7) 

4.2/1. 8 

Own  Data  Accessibility  (q.  8) 

5. 2/1.1 

5 

6 

7 

8 

% Problems  Correctly  Solved 

.41/. 32 

.79/. 28 

.62/. 46 

.94/. 20 

Solution  Time  (Min.) 

7. 8/3. 6 

5. 3/2. 8 

4. 3/9. 5 

2. 8/1. 8 

Problem  Difficulty  (q.  1) 

3. 3/1. 4 

2. 8/1. 4 

5. 0/1. 6 

2. 8/1.1 

Solution  Certainty  (q.  2)  * ** 

1*5. 4/1. 5 
3. 0/1.0 

3. 3/1. 5 

Problem  Fidelity  (q.  4) 

6. 1/1. 2 

5. 9/1. 3 

HR  Data  Utility  (q.  5) 

3. 1/1. 5 

3. 7/1. 5 

Own  Data  Utility  (q.  7) 

5.6/1. 3 

4. 7/1. 7 

Own  Data  Accessibility  (q.  8) 

5. 4/1. 7 

5. 4/1. 3 

*First  Number  = Mean  of  36  participants 
Second  Number  = Standard  Deviation  of  Mean 

**Data  from  problem  5 (above,  without  handbook,  below,  with 
handbook) . 


1 

l 
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Appendix  C (continued) 


Data  Category 

Problems 

9 

10 

11 

12 

% Problems  Correctly  Solved 

.56/. 48 

.70/. 37 

.92/. 25 

Solution  Time  (Min.) 

2. 1/6. 3 

4. 3/2. 7 

4. 3/2.0 

Problem  Difficulty  (q.  1) 

3. 8/1.4 

2. 3/0. 9 

3. 5/1. 5 

3. 0/1.0 

Solution  Certainty  (q.  2)  * 

*5. 3/1. 4 
4. 1/1. 4 

2. 7/1.1 

3. 4/0. 9 

Problem  Fidelity  (q.  4) 

5.6/1. 7 

5. 4/1. 5 

5. 5/1. 3 

HR  Data  Utility  (q.  5) 

4. 2/1. 6 

3. 6/1. 4 

3. 4/1. 6 

Own  Data  Utility  (q.  7) 

4. 4/1. 4 

4. 9/1. 5 

4.6/1. 3 

Own  Data  Accessibility  (q.  8) 

5. 2/1. 2 

5.2/1. 3 

5.2/1. 3 

**Data  from  problem  9 (above,  without  handbook,  below,  with 
handbook) . 


■fru.S.  GOVERNMENT  PRINTING  OFFICE:  1977  - 771-057/9 
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